Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are presented for examination.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on July 29th, 2024, and September 3rd, 2026, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6 and 7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitation “the cut layer” at the server neural network, “the cut layer” at the first local neural network, and “the cut layer” at the second local neural network. There is insufficient antecedent basis for this limitation in the claim. Claim 1 recites “a first cut layer data set” and “a second cut layer” at the worker local neural networks and does not recite a cut layer at the server neural network. It is unclear which cut layer “the cut layer” refers to and where the recited cut layer at the server neural network is established.
Claim 7 recites “until the error has reached an accepted level.” The term “accepted level” is a relative term of degree which renders the claim indefinite. The term “accepted level” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The specification does not provide metes and bounds for the claimed term.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 8-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over non-patent literature Ceballos ("SplitNN-driven Vertical Partitioning", 2020) in view of non-patent literature Thapa ("SplitFed: When Federated Learning Meets Split Learning", September 2021), and further in view of non-patent literature Jorgensen ("Conservative or Liberal? Personalized Differential Privacy", 2015).
Regarding claim 1,
Ceballos teaches [a] system for … split vertical learning, comprising: (Pages 1-2 Introduction of Ceballos, “…we are interested in the setting where multiple entities (clients) collaborate for a targeted task at hand, under the coordination of a central server or service provider. Each client’s raw data is stored locally and are not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective… The other variants to SplitNN for distributed private training can also be achieved with Federated Learning… the goal of our work is to learn a shared model using vertical partitioned data coming from several sources while preserving data privacy.”, Page 4 Section 4 Experiments, “We test our system experimentally on three financial datasets.”
Ceballos discloses a split-learning system operating on vertically partitioned data -i.e., split vertical learning)
a server node comprising … a server neural network (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network"
The shared deep servers-network that computes the last stage of the combined model corresponds to a server node including a server neural network.)
a first data silo in communication with the server node across a network, wherein the first data silo comprises a first subset of a dataset (Page 1 Introduction of Ceballos, "Each client's raw data is stored locally and are not exchanged or transferred", Page 4 Section 4, "We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other.", Page 3 Section 3, “In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network.”
Under broadest reasonable interpretation in light of the specification (e.g., Paragraphs 6 and 14, "various databases or silos hold a number of different columns of data" and that "each module lives in a data silo"), a data silo is construed as the local data storage holding a client's vertical feature subset (e.g., the bank client data). Ceballos's per-client local data store where each client's raw data is stored locally corresponds to the claimed data silo.)
a second data silo in communication with the server across the network, wherein the second data silo comprises a second subset of the dataset (Page 4 Section 4 of Ceballos, "We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other… Once again there is no coherent vertical split for the data so we choose to split the features arbitrarily into two sets.", Pages 3-4 Section 3 of Ceballos, "The full forward pass will be a concatenation of the forward passes coming from each client-network…", Page 3 Section 3, “In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network.”
Ceballos holds the complementary feature subset (the social and economic context attributes) in a second client's local storage. This local data storage corresponds to the claimed second data silo comprising a second subset.)
a first worker node implemented at the first data silo, wherein the first worker node comprises a first local neural network… wherein the first worker node is configured to produce a first [output] data set from the first data subset using the first local neural network (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network…”, Page 3 Figure 1 Caption, "Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest…", Pages 3-4 Section 3, “The full forward pass will be a concatenation of the forward passes coming from each client-network {F1 ++ F2 ++ . . . Fk}, k ∈ {1, . . . , K} where K is the maximum number of clients.”, See Figure 1,
PNG
media_image1.png
229
677
media_image1.png
Greyscale
Ceballos's client computing node (the first worker node) is implemented at that client's local data silo and runs a separate partial neural network on the silo's feature subset. The partial network corresponds to the first local neural network and its transmitted output corresponds to the first output data set.)
a second worker node implemented at the second data silo, wherein the second worker node comprises a second local neural network… wherein the second worker node is configured to produce a second [output] from the second data subset using the second local neural network (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side", Pages 3-4 Section 3 of Ceballos, "The full forward pass will be a concatenation of the forward passes coming from each client-network {F1 ++ F2 ++ … Fk}, k ∈ {1, … , K} where K is the maximum number of clients", Page 3 Section 3 of Ceballos, "each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network", See Figure 1.
Ceballos's client computing node (the second worker node) is implemented at the second client's local data silo and runs a separate partial neural network on the silo's feature subset. The partial network corresponds to the second local neural network and its transmitted output corresponds to the second output data set.)
wherein the server node is further configured to aggregate the first and second [output] data sets (Page 2 Section 3 of Ceballos, "To this end, we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side…", Pages 3-4 Section 3, “The full forward pass will be a concatenation of 3 the forward passes coming from each client-network {F1 ++ F2 ++ . . . Fk}”
The server-side network collects the outputs of the separate client-networks and combines them (e.g., by concatenation) before the last stage of the model.)
Ceballos does not teach a differentially private system and a server processor… the first worker node comprises a … neural network and a first optimization routine, wherein the first worker node is configured to produce a first cut layer from the first data subset using the … neural network… the first optimization routine is configured to apply a first parameterized level of noise to the first cut level to produce a first smashed data set… the second worker node comprises a … neural network and a second optimization routine, wherein the second worker node is configured to produce a second cut layer from the second data subset using the … neural network… and the second optimization routine is configured to apply a second parameterized level of noise to the second cut layer to produce a second smashed data set…train against the first and second smashed data sets for at least one additional layer in the server neural network to calculate an error against a prediction… and send a set of smashed gradients to the first and second worker nodes based on the error.
Thapa, in the same field of endeavor, teaches a system for differentially private… learning, comprising… the first worker node comprises a… neural network and a first optimization routine, wherein the first worker node is configured to produce a first cut layer from the… data [set] using the … neural network (Page 2 Section 2, “The communication involves sending activations, called smashed data, of the split layer, called cut layer, of the client-side network to the server, and receive the gradients of the smashed data from the server-side operations.”, Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer)”, Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)… EnsureClientBackprop(dAk,t): Back-propagation, calculate gradients k(WC k,t) with dAk,t 2-norm of each gradient is clipped and a calibrated noise is added based on Equation (2) and (3) to calculate ˜gk,t Update WC k,t ← WC k,t −η˜gk,t", Page 1 Introduction, “split learning (SL) was introduced. SL splits the full ML model into multiple smaller network portions and train them separately on a server, and distributed clients with their local data… Data privacy and model’s robustness are enhanced at the architectural level in SFL by the differential privacy-based measures … and PixelDP…”
Thapa's client-side network computes its data up to the split layer, and includes a noise layer as the component that operates on that layer's output. The split/cut layer of the client-side network corresponds to the first cut layer. Thapa's client-side training constitutes an optimization routine that minimizes the loss by stochastic gradient descent, clipping the gradients, adding calibrated noise, and updating the client-side model parameters. This client-side optimization routine corresponds to the claimed first optimization routine. Paragraph 15 of the instant specification describes the "optimization module … which applies noise during each parameter update iteration," and Paragraph 19 describes applying noise to the local gradients before updating the weights. Consistent with the specification, Thapa's client-side optimization routine (Algorithm 2) applies calibrated noise during its parameter-update iteration, corresponding to the claimed optimization routine. )
the first optimization routine is configured to apply a first parameterized level of noise to the first cut level to produce a first smashed data set (Page 5 Algorithm 2 of Thapa, "Noise layer: Perturbs the outputs of the layer L based on Equation (5)", Pages 5-6 Section 3.2 of Thapa, "...to avoid privacy leakage and further strengthen data privacy and model robustness against potential adversarial ML settings, we integrate a noise layer in the client-side model based on the concepts of PixelDP. This extended measure utilizes the noise application mechanism involved in differential privacy to add a calibrated noise to the output (e.g., activation vectors) of a layer at the client-side model while maintaining utility… forwarding private versions of the smashed data to the main server; hence, preserving the privacy of smashed data", Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)", Page 6 Equation 5, “Secondly, Laplacian noise with scale ∆IA p,q ε 0 is applied to randomize any data as follows:
PNG
media_image2.png
50
195
media_image2.png
Greyscale
”
Within the client-side optimization routine described above, a parameterized (privacy-budget ε) noise is applied to the cut-layer activation output, producing a private smashed version of that output. Applying the parameterized noise as part of the client-side optimization routine to produce the smashed data corresponds to the first optimization routine applying a first parameterized level of noise to the first cut level to produce a first smashed data set.)
the second worker node comprises a… neural network and a second optimization routine, wherein the second worker node is configured to produce a second cut layer from the … data [set] using the… neural network (Page 2 Section 2, “The communication involves sending activations, called smashed data, of the split layer, called cut layer, of the client-side network to the server, and receive the gradients of the smashed data from the server-side operations.”, Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer)”, Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)… EnsureClientBackprop(dAk,t): Back-propagation, calculate gradients k(WC k,t) with dAk,t 2-norm of each gradient is clipped and a calibrated noise is added based on Equation (2) and (3) to calculate ˜gk,t Update WC k,t ← WC k,t −η˜gk,t”, Page 1 Introduction, “split learning (SL) was introduced. SL splits the full ML model into multiple smaller network portions and train them separately on a server, and distributed clients with their local data.”
Since Thapa's client-side optimization routine (the stochastic-gradient-descent parameter update of Algorithm 2, including its noise application) runs on all clients in parallel, a second client likewise performs its own client-side optimization routine and computes to its own cut layer. The second client's SGD-based client-side training procedure corresponds to the claimed second optimization routine, and its split/cut layer corresponds to the second cut layer.)
and the second optimization routine is configured to apply a second parameterized level of noise to the second cut layer to produce a second smashed data set (Pages 5-6 Section 3.2 of Thapa, "...to avoid privacy leakage and further strengthen data privacy and model robustness against potential adversarial ML settings, we integrate a noise layer in the client-side model based on the concepts of PixelDP. This extended measure utilizes the noise application mechanism involved in differential privacy to add a calibrated noise to the output (e.g., activation vectors) of a layer at the client-side model while maintaining utility… forwarding private versions of the smashed data to the main server; hence, preserving the privacy of smashed data", Page 5 Algorithm 2 of Thapa, " /* Runs on Client k */… for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)", Page 3 Section 3.1, “All clients perform forward propagation on their client-side model in parallel, including its noise layer, and pass their smashed data to the main server…”, Page 6 Equation 5, “Secondly, Laplacian noise with scale ∆IA p,q ε 0 is applied to randomize any data as follows:
PNG
media_image2.png
50
195
media_image2.png
Greyscale
”
Within the second client's optimization routine, a parameterized (privacy-budget ε) noise is applied to its cut-layer activation output, producing a private smashed version. Applying the parameterized noise as part of the second client-side optimization routine to produce the smashed data corresponds to the second optimization routine applying a second parameterized level of noise to the second cut layer to produce a second smashed data set.)
train against the first and second smashed data sets for at least one additional layer in the server neural network to calculate an error against a prediction (Section 3.1 of Thapa, "the main server processes the forward propagation and back-propagation on its server-side model with each client's smashed data separately in … parallel.", Algorithm 1 of Thapa, "EnsureMainServer executes:for each client k ∈ St in parallel do… Forward propagation with Ak,t on WS t, compute Yˆ k… Loss calculation with Yk and Yˆ k Back-propagation calculate O`k(WS t ; AS t ) Send dAk,t := O`k(AS t ;WS t ) (i.e., gradient of the Ak,t) to client k for ClientBackprop(dAk,t)"
Thapa's main server runs the server-side network (WS) on each client's smashed data, computes a predicted label Yˆk, and calculates the loss between the true label Yk and the predicted label Yˆk. Processing the smashed data through the server-side network to compute a loss against the predicted label corresponds to training against the data sets for at least one additional layer in the server neural network to calculate an error against a prediction.)
and send a set of smashed gradients to the first and second worker nodes based on the error (Page 3 Section 3.1 of Thapa, "It then sends the gradients of the smashed data to the respective clients for their back-propagation", Algorithm 1 of Thapa, "EnsureMainServer executes: for each client k ∈ St in parallel do… Forward propagation with Ak,t on WS t, compute Yˆ k… Loss calculation with Yk and Yˆ k Back-propagation calculate O`k(WS t ; AS t ) Send dAk,t := O`k(AS t ;WS t ) (i.e., gradient of the Ak,t) to client k for ClientBackprop(dAk,t)"
After computing the loss, Thapa's server back-propagates and returns the gradients of the smashed data (dAk,t) to each respective client for its local back-propagation. Returning these smashed-data gradients to the clients based on the computed loss corresponds to sending a set of smashed gradients to the first and second worker nodes based on the error.)
a server processor (Page 3 Section 3.1 of Thapa, "the main server can be a cloud server or a researcher with high-performance computing resources", Page 7 Section 4 of Thapa, "we use the High-Performance Computing (HPC) platform that is built on Dell EMC's PowerEdge platform with partner GPUs for computation … We run clients and servers on different computing nodes of the cluster … one GPU (Tesla P100-SXM2-16GB)…"
Thapa's main server executes the server-side network on high-performance computing hardware including a GPU.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's vertically-partitioned split-learning architecture with Thapa's differentially-private noise layer applied to the cut-layer output, in order to forward only a private version of the smashed data to the server and thereby preserve the privacy of the smashed data against reconstruction from the cut-layer activations (Section 3.2 of Thapa).
Ceballos in view of Thapa does not teach that the second parameterized level of noise is set independently from the first.
Jorgensen, in the same field of endeavor, teaches wherein the second parameterized level of [privacy] is set independently from the first parameterized level of [privacy] (Page 3 Section II B, “Users specify a (possibly non-uniform) valuation on their privacy that expresses their privacy cost incurred by participating in an e-differentially private analysis (as a function of e), and hence the amount of compensation due if their data is used.”, Page 4 Section III, "PDP makes use of a privacy specification, in which each user in U independently specifies the privacy requirement for their data… a mapping... from users to personal privacy preferences, where a smaller value represents a stronger privacy preference", Page 6 Example 1 Section IV, “...conservative users have a strong privacy requirement, say εc = 0.1, and liberal users have a relatively weak requirement, say εL = 1".”
Jorgensen's personalized differential privacy lets each participant independently specify its own privacy parameter (ε), and different participants take different, independently set values (e.g., 0.1 versus 1).)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos in view of Thapa's per-worker noise layers with Jorgensen's per-participant privacy specification so that each worker's privacy level is set independently in order to guarantee each data silo precisely its own required level of privacy rather than forcing the most conservative level across all silos and allow users/clients to have control over their data (Section I and Section IV of Jorgensen).
Regarding claim 2,
Ceballos does not teach the first parameterized level of noise and the second parameterized level of noise.
Thapa teaches the first parameterized level of noise and the second parameterized level of noise (Page 5 Algorithm 2 of Thapa, " /* Runs on Client k */… for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)", Page 6 Equation 5, “Secondly, Laplacian noise with scale ∆IA p,q ε 0 is applied to randomize any data as follows:
PNG
media_image2.png
50
195
media_image2.png
Greyscale
”
Thapa's noise layer adds a privacy-budget-parameterized differential-privacy noise to the cut-layer activation output and this noise layer runs on every client in parallel. The first client's noise layer applying its parameterized noise corresponds to the first parameterized level of noise, and the second client's noise layer applying its parameterized noise corresponds to the second parameterized level of noise.)
Ceballos in view of Thapa does not teach that the noise levels have different values.
Jorgensen, in the same field of endeavor, teaches wherein the first parameterized level of [privacy] has a different value from the second parameterized level of [privacy] (Page 4 Section III of Jorgensen, "each user in U independently specifies the privacy requirement for their data… a mapping... from users to personal privacy preferences, where a smaller value represents a stronger privacy preference", Page 6 Example 1 Section IV, "conservative users have a strong privacy requirement, say [εc] = 0.1, and liberal users have a relatively weak requirement, say [εL] = 1."
Jorgensen's independently-specified privacy parameters take different values for different participants (e.g., 0.1 versus 1).) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos in view of Thapa's per-worker noise layers with Jorgensen's per-participant privacy specification so that the two silos' noise levels take different values in order allow users/clients to have control over their data (Section I of Jorgensen).
Regarding claim 3,
Ceballos teaches wherein the first subset of the dataset comprises a first vertical split of the dataset, and the second subset of the dataset comprises a second vertical split of the dataset (Page 4 Section 4 of Ceballos, "distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other… we choose to split the features arbitrarily into two sets"
Ceballos partitions the feature columns vertically into distinct splits held by different clients (e.g., bank-client features in one split, social/economic features in the other), corresponding to the claimed first and second vertical splits.)
Regarding claim 4,
Ceballos teaches wherein the first worker node prevents access to the first subset of the dataset by the server node, and further wherein the second worker node prevents access to the second subset of the dataset by the server node (Abstract of Ceballos, "SplitNN does not share raw data or model details with collaborating institutions", Page 1 Introduction of Ceballos, "Each client's raw data is stored locally and are not exchanged or transferred", Figure 1 and its Caption, “Vertical SplitNN architecture: Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest and performs back-propagation.”
Since each client's raw feature subset is stored locally and never exchanged, the server node is prevented from accessing either subset.)
Regarding claim 8,
Ceballos teaches [a] method for … split vertical learning, the method comprising the steps of (Pages 1-2 Introduction of Ceballos, “Motivated by the above observations, we are interested in the setting where multiple entities (clients) collaborate for a targeted task at hand, under the coordination of a central server or service provider. Each client’s raw data is stored locally and are not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective… The other variants to SplitNN for distributed private training can also be achieved with Federated Learning… the goal of our work is to learn a shared model using vertical partitioned data coming from several sources while preserving data privacy.” Ceballos discloses the split-vertical-learning method operating on vertically partitioned data.)
initializing a first worker node at a first data silo comprising a first data slice of a raw dataset, wherein the first worker node comprises a first local neural network receiving as input the first data slice (Page 4 Section 4, "We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other.", Page 3 Section 3, “In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network.”, Page 1 Introduction of Ceballos, "Each client's raw data is stored locally and are not exchanged or transferred", Pages 3-4 Section 3 of Ceballos, "The full forward pass will be a concatenation of the forward passes coming from each client-network…", Pages 2-3 Section 3 of Ceballos, "...we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network"
Ceballos runs a separate partial neural network at each client, each receiving as input its own vertical slice of the data, corresponding to the claimed initialization of first worker nodes with a first local neural network.),
and initializing a second worker node at a second data silo comprising a second data slice of the raw dataset, wherein the second worker node comprises a second local neural network receiving as input the second data slice (Page 4 Section 4, "We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other.", Page 3 Section 3, “In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network.”, Page 1 Introduction of Ceballos, "Each client's raw data is stored locally and are not exchanged or transferred", Pages 3-4 Section 3 of Ceballos, "The full forward pass will be a concatenation of the forward passes coming from each client-network…", Pages 2-3 Section 3 of Ceballos, "...we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network"
Ceballos runs a separate partial neural network at each client, each receiving as input its own vertical slice of the data, corresponding to the claimed initialization of a second worker node with a second local neural network.)
training the first local neural network against the first data slice … to produce a first local neural network output, and training the second local neural network against the second data slice … to produce a second local neural network output (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… In our method each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network…”, Page 3 Figure 1 Caption, "Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest…", Pages 3-4 Section 3, “The full forward pass will be a concatenation of the forward passes coming from each client-network {F1 ++ F2 ++ . . . Fk}, k ∈ {1, . . . , K} where K is the maximum number of clients.”, See Figure 1,
PNG
media_image1.png
229
677
media_image1.png
Greyscale
Each partial network is trained on its slice and produces an output that is transmitted to the server, corresponding to the claimed first and second local neural network outputs.)
and at a server node comprising an aggregate neural network, aggregating the first [output] data set and second [output] data set. (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side", Pages 3-4 Section 3 of Ceballos, "The full forward pass will be a concatenation of the forward passes coming from each client-network {F1 ++ F2 ++ … Fk}, k ∈ {1, … , K} where K is the maximum number of clients", Page 3 Section 3 of Ceballos, "each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network", See Figure 1.
The server-side network aggregates (e.g., concatenates) the workers' outputs before the last stage.)
Ceballos does not teach explicitly a method for… differentially private … learning… that the output is produced up to a cut layer, nor applying a noise level to the output to produce a smashed data set, nor applying a noise level to local gradients to produce weights… at a first optimization routine at the first worker node, applying a first noise level to the first local neural network output to produce a first smashed data set, and applying the first noise level to a first set of local gradients to produce a first set of weights…at a second optimization routine at the second worker node, applying a second noise level to the second local neural network output to produce a second smashed data set, and applying the second noise level to a second set of local gradients to produce a second set of weights… wherein the second optimization routine operates independently from the first optimization routine
Thapa, in the same field of endeavor, teaches a method for… differentially private … learning… training the … neural network against the first data slice up to a cut layer (Page 2 Section 2 of Thapa, "The communication involves sending activations, called smashed data, of the split layer, called cut layer, of the client-side network to the server", Page 4 Section 3.2 of Thapa, "The main server has access only to the smashed data (i.e., activation vectors of the cut layer)", Page 1 Introduction, “Our contributions are mainly two-fold: Firstly, we are the first to propose SFL. Data privacy and model’s robustness are enhanced at the architectural level in SFL by the differential privacy-based measures [1] and PixelDP [18].”, Page 5 Section 3.2, “the final convergence will exhibit a privacy level of (ε, δ)- differential privacy, where (ε, δ) is the overall privacy cost of the client-side model.”
Thapa's client-side network computes its data up to the split layer, corresponding to the claimed cut layer at which the local neural network output is produced. Thapa's split-learning architecture incorporates differential privacy such that the trained model satisfies (ε, δ)-differential privacy, corresponding to the claimed differentially private learning.)
at a first optimization routine at the first worker node, applying a first noise level to the first local neural network output to produce a first smashed data set, and applying the first noise level to a first set of local gradients to produce a first set of weights (Pages 5-6 Section 3.2 of Thapa, "...to avoid privacy leakage and further strengthen data privacy and model robustness against potential adversarial ML settings, we integrate a noise layer in the client-side model based on the concepts of PixelDP. This extended measure utilizes the noise application mechanism involved in differential privacy to add a calibrated noise to the output (e.g., activation vectors) of a layer at the client-side model while maintaining utility… forwarding private versions of the smashed data to the main server; hence, preserving the privacy of smashed data", Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)… EnsureClientBackprop(dAk,t): Back-propagation, calculate gradients k(WC k,t) with dAk,t 2-norm of each gradient is clipped and a calibrated noise is added based on Equation (2) and (3) to calculate ˜gk,t Update WC k,t ← WC k,t −η˜gk,t", Page 6 Equation 5, “Secondly, Laplacian noise with scale ∆IA p,q ε 0 is applied to randomize any data as follows:
PNG
media_image2.png
50
195
media_image2.png
Greyscale
”, Page 5 Algorithm 2 of Thapa, "`2-norm of each gradient is clipped and a calibrated noise is added based on Equation (2) and (3) to calculate g˜k,t… Update WC k,t ← WC k,t − ηg˜k,t"
Thapa's noise layer applies a privacy-budget-parameterized noise to the cut-layer output to produce the smashed data, and separately clips and adds calibrated noise to the local gradients before updating the client-side weights. Thapa's client-side training constitutes an optimization routine that minimizes the loss by stochastic gradient descent, clipping the gradients, adding calibrated noise, and updating the client-side model parameters. This client-side optimization routine corresponds to the claimed first optimization routine. Paragraph 15 of the instant specification describes the "optimization module … which applies noise during each parameter update iteration," and Paragraph 19 describes applying noise to the local gradients before updating the weights. Consistent with the specification, Thapa's client-side optimization routine (Algorithm 2) applies calibrated noise during its parameter-update iteration, corresponding to the claimed optimization routine.)
at a second optimization routine at the second worker node, applying a second noise level to the second local neural network output to produce a second smashed data set, and applying the second noise level to a second set of local gradients to produce a second set of weights, (Pages 5-6 Section 3.2 of Thapa, "...to avoid privacy leakage and further strengthen data privacy and model robustness against potential adversarial ML settings, we integrate a noise layer in the client-side model based on the concepts of PixelDP. This extended measure utilizes the noise application mechanism involved in differential privacy to add a calibrated noise to the output (e.g., activation vectors) of a layer at the client-side model while maintaining utility… forwarding private versions of the smashed data to the main server; hence, preserving the privacy of smashed data", Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t… Noise layer: Perturbs the outputs of the layer L based on Equation (5)… EnsureClientBackprop(dAk,t): Back-propagation, calculate gradients k(WC k,t) with dAk,t 2-norm of each gradient is clipped and a calibrated noise is added based on Equation (2) and (3) to calculate ˜gk,t Update WC k,t ← WC k,t −η˜gk,t", Page 3 Section 3.1, “All clients perform forward propagation on their client-side model in parallel, including its noise layer, and pass their smashed data to the main server…”, Page 6 Equation 5, “Secondly, Laplacian noise with scale ∆IA p,q ε 0 is applied to randomize any data as follows:
PNG
media_image2.png
50
195
media_image2.png
Greyscale
”
Since Thapa's client-side optimization routine (the stochastic-gradient-descent parameter update of Algorithm 2, including its noise application) runs on all clients in parallel, a second client likewise performs its own client-side optimization routine and computes to its own cut layer. The second client's SGD-based client-side training procedure corresponds to the claimed second optimization routine, and its split/cut layer corresponds to the second cut layer.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's split-vertical training with Thapa's dual differential-privacy perturbation of both the cut-layer output and the local gradients in order to obtain a differentially-private client-side model that guarantees differential privacy while limiting leakage from both the transmitted activations and the model updates (Section 3.2 of Thapa).
Ceballos in view of Thapa does not teach that the second optimization routine operates independently from the first optimization routine.
Jorgensen, in the same field of endeavor, teaches wherein the second [privacy] operates independently from the [privacy] (Abstract of Jorgensen, "a generalization of differential privacy in which users specify a personal privacy requirement for their data", Section III of Jorgensen, "PDP makes use of a privacy specification, in which each user in U independently specifies the privacy requirement for their data", Definition 5 of Jorgensen, "a mapping... from users to personal privacy preferences, where a smaller value represents a stronger privacy preference", Example 1 Section IV, “...conservative users have a strong privacy requirement, say [εc] = 0.1, and liberal users have a relatively weak requirement, say [εL] = 1".”
Jorgensen's personalized differential privacy lets each participant independently specify its own privacy parameter (ε), and different participants take different, independently-set values (e.g., 0.1 versus 1).)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos in view of Thapa's per-worker noise layers with Jorgensen's per-participant privacy specification so that each worker's privacy level is set independently in order to guarantee each data silo precisely its own required level of privacy rather than forcing the most conservative level across all silos (Section I and Section IV of Jorgensen).
Claim 9 recites similar limitations to claim 2. Therefore, claim 9 is rejected using the same rationale as claim 2.
Regarding claim 10,
Ceballos teaches training against the aggregated first [output] data set and the second [output] data set at the aggregate neural network (Pages 2-3 Section 3 of Ceballos, “we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1… All these setups require communication on every iteration since they are jointly optimized by back-propagating the error from the main network to the smaller ones.”, Page 4 Section 3 of Ceballos, "All these setups require communication on every iteration since they are jointly optimized by back-propagating the error from the main network to the smaller ones."
The server-side network trains on the aggregated worker outputs at its last stage, corresponding to training against the aggregated data sets at the aggregate neural network.)
Ceballos does not teach explicitly training against the … first smashed data set and the second smashed data set at the … neural network to calculate an error against a prediction.
Thapa, in the same field of endeavor, teaches training against the … first smashed data set and the second smashed data set at the … neural network to calculate an error against a prediction (Page 4 Algorithm 1 of Thapa, "Forward propagation with Ak,t on WS t, compute Yˆ k… Loss calculation with Yk and Yˆ k”, Page 3 Section 3.1, “All clients perform forward propagation on their client-side model in parallel, including its noise layer, and pass their smashed data to the main server…”, Page 2 Section 2, “The communication involves sending activations, called smashed data, of the split layer, called cut layer, of the client-side network to the server, and receive the gradients of the smashed data from the server-side operations.”, Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer)”
Thapa's main server computes a predicted label Yˆk and calculates the loss between the true label Yk and the predicted label, corresponding to calculating an error against a prediction.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's server-side aggregation with Thapa's server-side loss computation against the predicted label in order to drive the iterative optimization of the aggregated split network toward the prediction target (Algorithm 1 of Thapa).
Regarding claim 11,
Ceballos does not teach the step of sending from the server node to the first worker node and the second worker node a set of updated parameters based on the error.
Thapa, in the same field of endeavor, teaches sending from the server node to the first worker node and the second worker node a set of updated parameters based on the error (Page 3 Section 3.1 of Thapa, "Then the main server processes the forward propagation and back-propagation on its server-side model with each client’s smashed data separately in (somewhat) parallel. It then sends the gradients of the smashed data to the respective clients for their back-propagation", Page 4 Algorithm 1 of Thapa, "for each client k ∈ St in parallel do… Send dAk,t := O`k(AS t ;WS t ) (i.e., gradient of the Ak,t) to client k for ClientBackprop(dAk,t)"
After computing the loss, the server returns the error-derived gradients of the smashed data to each respective client for its update.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos’s teaching with Thapa's return of the error-derived parameters to the respective workers in order to enable each worker to complete its local back-propagation and update its client-side model (Section 3.1 of Thapa).
Regarding claim 12,
Ceballos teaches wherein the first data slice is a first vertical slice of the dataset, and the second data slice is a second vertical slice of the data set that does not overlap with the first vertical slice of the dataset (Page 4 Section 4, “Once again there is no coherent vertical split for the data so we choose to split the features arbitrarily into two sets… We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other.” Ceballos partitions the features vertically into two distinct, non-overlapping sets held by different clients, corresponding to the claimed first and second non-overlapping vertical slices.)
Regarding claim 13,
Ceballos teaches blocking access by the server node to the first data slice at the first worker node and blocking access by the server node to the second data slice at the second worker node (Abstract of Ceballos, "SplitNN does not share raw data or model details with collaborating institutions", Page 1 Introduction of Ceballos, "Motivated by the above observations, we are interested in the setting where multiple entities (clients) collaborate for a targeted task at hand, under the coordination of a central server or service provider. Each client's raw data is stored locally and are not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.", Figure 1 and its Caption, “Vertical SplitNN architecture: Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest and performs back-propagation.”
Since each client's raw feature subset is stored locally and never exchanged, the server node is prevented from accessing either subset.).
Regarding claim 14,
Ceballos teaches the step of back-propagating the error at the aggregate neural network until the … layer is reached (Page 3 Figure 1 Caption of Ceballos, "the server which computes the rest and performs back-propagation and returns back the jacobians to the client which can perform their respective back-propagation", Page 3 Section 3, “All these setups require communication on every iteration since they are jointly optimized by back-propagating the error from the main network to the smaller ones… F(X) ← LN (LN−1...(L0(X))) Gradients can be backpropagated over each layer to generate gradients of previous layers and to update the current layer. We will use L T i (gradient) to denote the process of backpropagation over one layer and F T (gradient) to denote backpropagation over the entire neural network.”
The server-side network back-propagates the error through its layers down to the boundary layer at which it returns the gradients (jacobians) to the clients. This boundary layer at which the server's back-propagation terminates and returns gradients corresponds to the claimed layer that is reached.)
Ceballos does not teach the cut layer.
Thapa, in the same field of endeavor, teaches the cut layer (Page 2 Section 2 of Thapa, "the split layer, called cut layer, of the client-side network to the server", Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer).”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's split-vertical training with Thapa's dual differential-privacy perturbation of both the cut-layer output and the local gradients in order to obtain a differentially-private client-side model that guarantees differential privacy while limiting leakage from both the transmitted activations and the model updates (Section 3.2 of Thapa).
Regarding claim 15,
Ceballos teaches the steps of back-propagating the error at the first local neural network beginning from the … layer, and back-propagating the error at the second local neural network beginning from the … layer. (Page 3 Figure 1 Caption, "Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest and performs back-propagation and returns back the jacobians to the client which can perform their respective back-propagation.", Pages 3-4 Section 3, “In the same way, the gradients on the concatenation layer will be {L1 ++ L2 ++ . . . Lk}. Thus, we only need to split this gradients before passing them to the upstream clients.”, Page 2 Section 3, “we propose to train partial neural networks (NN) on each client and then aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side, as seen in Figure 1.”
The server returns the boundary-layer gradients (jacobians) to each client, and each client then performs its own back-propagation through its local layers beginning from that returned boundary-layer gradient. Each worker back-propagating from the returned boundary-layer gradient corresponds to the claimed back-propagating at the first and second local neural networks beginning from that layer.)
Ceballos does not teach the cut layer.
Thapa, in the same field of endeavor, teaches the cut layer (Page 2 Section 2 of Thapa, "the split layer, called cut layer, of the client-side network to the server", Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer).”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's server-side back-propagation to the gradient-return boundary with Thapa's designation of that boundary as the cut layer in order to identify the split-layer boundary at which the server returns gradients to the workers (Section 2 of Thapa).
Regarding claim 16,
Ceballos teaches the first local neural network and the second local neural network (Pages 2-3 Section 3 of Ceballos, "we propose to train partial neural networks (NN) on each client", Page 3 Section 3, "each partial clients-network encodes its data into a different space and then transmits it to train a shared deep servers-network"
Ceballos runs a separate partial neural network at each client on that client's own vertical feature slice, providing the claimed first and second local neural networks.)
Ceballos does not teach explicitly the step of, after back-propagating from the error, feeding forward at each local neural network up to the cut layer.
Thapa, in the same field of endeavor, teaches the step of, after back-propagating from the error at the … neural network, feeding forward at … neural network up to the cut layer, and further comprising the step of, after back-propagating from the error at the … neural network, feeding forward at the … neural network up to the cut layer (Page 2 Section 2 of Thapa, "the split layer, called cut layer, of the client-side network to the server", Page 5 Algorithm 2 of Thapa, "for each local epoch e from 1 to E do… Forward propagation with data Xk up to a layer L ≥ 1 in WC k,t", Page 5 Section 3.2 of Thapa, "We apply calibrated noise iteratively until the model converges or reaches a specified number of iterations"
Thapa's client-side network iterates per epoch, so after each backward pass the client-side network again feeds forward up to the split layer. Applying this iterative feed forward to the cut layer step to each of Ceballos's two per-silo local neural networks yields the claimed feed-forward step at the first and second local neural networks up to the cut layer.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's two per-silo local neural networks with Thapa's iterative feed-forward-to-the-cut-layer training step in order to continue training each worker's local network across successive iterations (Section 3.2 of Thapa).
Regarding claim 17,
Ceballos does not teach the step of producing a subsequent first smashed data set and a subsequent second smashed data set.
Thapa, in the same field of endeavor, teaches the step of producing a subsequent first smashed data set and a subsequent second smashed data set (Page 5 Algorithm 2 of Thapa, " /* Runs on Client k */ … With the output from the noise layer, continue forward propagation to the remaining layers of WC k,t, and get the activations of its final layer Ak,t (smashed data)", Page 5 Section 3.2 of Thapa, "We apply calibrated noise iteratively until the model converges or reaches a specified number of iterations", Page 3 Section 3.1 of Thapa, "All clients perform forward propagation on their client-side model in parallel, including its noise layer, and pass their smashed data to the main server"
On each iteration, each of the clients again produces a noised cut-layer output (smashed data), corresponding to the claimed subsequent first and second smashed data sets.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's split-vertical architecture with Thapa's iterative production of smashed data across epochs in order to supply the server with updated smashed data (Section 3.2 of Thapa).
Regarding claim 18,
Thapa teaches calculating a second error at the server node after receiving the subsequent first smashed data set and subsequent second smashed data set (Page 4 Algorithm 1 of Thapa, “Notations: (1) St is a set of K clients at t time instance, (2) Ak,t is the smashed data of client k at t, (3) Yk and ˆ Yk are the true and predicted labels, respectively, of the client k… EnsureMainServer executes… for each client k ∈ St in parallel do … Forward propagation with Ak,t on WS t, compute Yˆ k… Loss calculation with Yk and Yˆ k… Back-propagation calculate k(WS t;AS t)", Page 5 Section 3.2, "We apply calibrated noise iteratively until the model converges or reaches a specified number of iterations.”
On each iteration the server recomputes the loss from the newly received smashed data, corresponding to the claimed calculation of a second error.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's teaching with Thapa's iterative recomputation of the loss on each round of received smashed data in order to continue optimizing the model until convergence (Section 3.2 of Thapa).
Regarding claim 20,
Jorgensen teaches setting the first noise level based on a first privacy epsilon value for the first data slice, and setting the second noise level based on a second privacy epsilon value for the second data slice (Definition 5 of Jorgensen, "a mapping... from users to personal privacy preferences, where a smaller value represents a stronger privacy preference"; Section III of Jorgensen, "each user in U independently specifies the privacy requirement for their data"; Example 1 Section IV of Jorgensen, "conservative users have a strong privacy requirement, say [εc] = 0.1, and liberal users have a relatively weak requirement, say [εL] = 1"
Jorgensen sets each participant's privacy via that participant's own epsilon value.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos-in-view-of-Thapa's per-worker noise with Jorgensen's per-participant epsilon values, in order to calibrate each data slice's noise to that slice's own privacy requirement (Section I of Jorgensen).
Claims 5-7 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ceballos ("SplitNN-driven Vertical Partitioning", 2020) in view of Thapa ("SplitFed: When Federated Learning Meets Split Learning", September 2021), in view of Jorgensen ("Conservative or Liberal? Personalized Differential Privacy", 2015), and in further view of Gharibi (US 20220029971 A1).
Regarding claim 5,
Ceballos teaches wherein the [server] is configured to transmit a set of specified vertical columns (Page 1 Introduction, “we are interested in the setting where multiple entities (clients) collaborate for a targeted task at hand, under the coordination of a central server or service provider. Each client’s raw data is stored locally and are not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.”, Page 4 Section 4 of Ceballos, "We use all the 16 feature dimensions for prediction and distribute them vertically among the clients. The vertical split is done based on the source of the features, with the bank client data in one split and all the social and economic context attributes in the other.”, Page 3 Figure 1 Caption, “Vertical SplitNN architecture: Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest and performs back-propagation.” Ceballos operates on a set of specified vertical feature columns designated for the training run. The server performs the computation of the vertical columns data on the clients and returns the jacobians to the client to perform back-propagation.)
Ceballos teaches and to return from the server node an aggregate view of the server neural network after training (Page 4 Section 4 of Ceballos, "we compare the results of training a centralized model (M) with training several split models (M1, M2, M3...), merging their outputs and using those as input for M", Page 8 Conclusion of Ceballos, "the proposed methods to merge the outputs of the split networks result in a shared model that performs on par with the centralized model.", See Figure 1.
Ceballos merges the split networks into a single shared model equivalent to a centralized model, corresponding to the claimed aggregate view of the server neural network returned after training.)
Ceballos in view of Thapa in view of Jorgensen does not teach an application programming interface (API) in communication with the server node.
Gharibi, in the same field of endeavor, teaches an application programming interface (API) in communication with the [party] node (Paragraph 77 of Gharibi, "A secure multi-party computation application programming interface (API) 310 can be used to communicate between the various parties 318, 314", Paragraph 77 of Gharibi, "A software development kit (SDK) 322 can provide instructions and libraries to the respective access points 302, 304, to interface with the API 310", Paragraph 86, “These models (also referenced as models 192, 194 in FIG. 1D) can be transmitted to a node or a node could be used such as the router 312 in FIG. 3 to cause the data to be exchanged via an API 310 for generating an average model (which may or may not be weighted) which can then be distributed across the varies clients as disclosed herein.”
Gharibi provides an API through which the parties communicate with and access the split/federated model on the server side, corresponding to the claimed API in communication with the server node.) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos in view of Thapa in view of Jorgensen's server node with Gharibi's application programming interface in order to provide a standardized interface through which collaborating parties interact with the training system and obtain the trained model (Paragraph 77 of Gharibi).
Regarding claim 6,
Ceballos teaches wherein the server node is further configured to back-propagate the error until the … layer at the server neural network, wherein the first worker node is configured to further back-propagate past the … layer at the first local neural network, and wherein the second worker node is configured to further back-propagate past the … layer at the second local neural network (Page 3 Figure 1 Caption, " Vertical SplitNN architecture: Each client computes a fixed portion of the computation graph and passes it to the server which computes the rest and performs back-propagation and returns back the jacobians to the client which can perform their respective back-propagation."
The server back-propagates the error to the cut layer and returns the gradient, and each worker then back-propagates through its own local layers.)
Ceballos does not teach the cut layer.
Thapa, in the same field of endeavor, teaches the cut layer (Page 2 Section 2 of Thapa, "the split layer, called cut layer, of the client-side network to the server", Page 4 Section 3.2, “The main server has access only to the smashed data (i.e., activation vectors of the cut layer).”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's server-side back-propagation to the gradient-return boundary with Thapa's designation of that boundary as the cut layer in order to identify the split-layer boundary at which the server returns gradients to the workers (Section 2 of Thapa).
Regarding claim 7,
Ceballos teaches the server node is further configured to repeatedly aggregate the first and second [output] data sets (Pages 2-3 Section 3 of Ceballos, "aggregate all of their outputs before feeding them to the last stage of the combined model on the server-side"; Page 3 Section 3 of Ceballos, "All these setups require communication on every iteration since they are jointly optimized by back-propagating the error from the main network to the smaller ones"
The server-side network combines the workers outputs on every iteration before the last stage of the model.)
Ceballos does not teach the smashed data sets and calculates the error against the prediction until the error has reached an accepted level.
Thapa, in the same field of endeavor, teaches the first and second smashed data sets and calculate the error against the prediction until the error has reached an accepted level (Page 5 Section 3.2 of Thapa, "We apply calibrated noise iteratively until the model converges or reaches a specified number of iterations", Page 5 Algorithm 2, “/* Runs on Client k */…”, Page 4 Algorithm 1, " Notations: (1) St is a set of K clients at t time instance, (2) Ak,t is the smashed data of client k at t, (3) Yk and ˆ Yk are the true and predicted labels, respectively, of the client k, (4) k is the gradient of the loss for the client k… for each client k ∈ St in parallel do… (Ak,t,Yk) ← ClientUpdate(WC k,t) … Forward propagation with Ak,t on WS t, compute ˆ Yk… Loss calculation with Yk and Yˆ k… Server-side model update…", Page 3 SFL work flow, “All clients perform forward propagation on their client-side model in parallel, including its noise layer, and pass their smashed data to the main server. Then the main server processes the forward propagation and back-propagation on its server-side model with each client’s smashed data separately in (somewhat) parallel. It then sends the gradients of the smashed data to the respective clients for their back-propagation. Afterward, the server updates its model by FedAvg, i.e., weighted averaging of gradients that it computes during the back-propagation on each client’s smashed data.”
Thapa's main server computes a predicted label Yˆk and calculates the loss between the true label Yk and the predicted label, iterating until the model converges.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos's repeated server-side aggregation of the worker outputs with Thapa's computation of the error against the predicted label iterated until convergence in order to repeatedly optimize the aggregated split network (Algorithm 1 and Section 3.2 of Thapa).
Regarding claim 19,
Ceballos teaches outputting a monolithic view of the aggregated neural network (Page 4 Section 4 of Ceballos, "we compare the results of training a centralized model (M) with training several split models (M1, M2, M3...), merging their outputs and using those as input for M", Page 8 Conclusion, "the proposed methods to merge the outputs of the split networks result in a shared model that performs on par with the centralized model", See Figure 1.
Ceballos merges the split networks into a single shared model, corresponding to the claimed monolithic view.)
Ceballos does not teach outputting that view from an application programming interface (API).
Gharibi, in the same field of endeavor, teaches [transmitting models to a node] …from an application programming interface (API) (Paragraph 77 of Gharibi, "A secure multi-party computation application programming interface (API) 310 can be used to communicate between the various parties 318, 314", Paragraph 77 of Gharibi, "A software development kit (SDK) 322 can provide instructions and libraries to the respective access points 302, 304, to interface with the API 310", Paragraph 86, “These models (also referenced as models 192, 194 in FIG. 1D) can be transmitted to a node or a node could be used such as the router 312 in FIG. 3 to cause the data to be exchanged via an API 310 for generating an average model…”
Gharibi provides an API through which the parties obtain the model.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to combine Ceballos in view of Thapa in view of Jorgensen's merged monolithic model with Gharibi's application programming interface in order to provide a standardized interface through which collaborating parties obtain the trained aggregated model (Paragraph 77 of Gharibi).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAJD MAHER HADDAD whose telephone number is (571)272-2265. The examiner can normally be reached Mon-Friday 8-5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.M.H./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
