CTFR 17/081,779 CTFR 98701 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Response to Amendment The amendments and arguments filed 02/10/2026 have been entered. Claims 1, 3-10 and 12-22 remain pending in the application. Applicant’s argument, with respect to 35 U.S.C 103 claim rejections filed 11/10/2025 have been fully considered but they are not persuasive. Therefore, the rejections as set forth in the previous office action will be maintained. The applicant argues that the cited combination fails to teach or suggest amended independent claims 1, 10 and 19. In particular, Applicant contends that Zhu teaches decentralized distributed/federated learning with local model parameters, local gradients updates, and neighbor-client communication, but does not teach the claimed GAN-based neural network, the first and second round of exchanging weights, generating an average weight at each node based on the exchanged weights sets, or updating the weight sets using a decentralized parallel OSG algorithm based on gradients and the average weight. Applicant further argues that Hardy does not cure these deficiencies because Hardy’s FL-GAN relies on workers sending parameters to a server for averaging, rather than the claimed decentralized node-to-node weight exchange and node-level average-weight generation. Applicant also argues that Lian teaches decentralized parallel stochastic-gradient descent, including neighbor exchange and averaging of local variables, but does not discloses updating the claimed sets of weight via an OSG algorithm or optimizing the amended non-concave, non-convex min/max loss function with provable non-asymptotic convergence to a first-order stationary point. Applicant further asserts that Mokhtari is directed extra-gradient/optimistic-gradient in bilinear, convex-concave saddle point setting, and therefore does not teach the amended non-concave/non-convex min/max loss function limitation. Applicant additionally argues that Voinea and Sanjabi do not cure the alleged deficiencies of the base combination, even though Sanjabi discusses non-convex, non-concave min-max and first-order stationary points. Accordingly, Applicant submits that the independent claims, and the dependent claims by virtue of their dependency, are patentable over the cited combinations. The examiner respectfully disagrees. Applicants’ arguments are not persuasive because they address the references individually and do not fully consider the combined teachings relied upon in the rejection. Zhu is relied upon for the decentralized federated learning architecture, including client nodes communicating with neighboring clients, exchanging local model parameters/weights, computing gradients, aggregating received parameters by weighted average, and updating local model parameters. Hardy is relied upon for teaching implementation of the neural network as part of a GAN in a federated/distributed training environment. Lian is relied upon for the decentralized parallel stochastic-gradient aspect. Mokhtari is relied upon for the optimistic gradient update technique, and Sanjabi is relied upon for the non-convex/non-concave min/max objective/loss function convergence limitation. Thus, the rejection does not require any single reference to teach every claimed feature. With respect to Applicant’s argument that Zhu does not teach the claimed rounds of exchanging weights or generating an average weight at each node, the argument is not persuasive. Zhu teaches exchanging local model parameter/weights with neighboring clients during repeated training rounds and aggregating received local model parameters using a weighted sum or weighted average. Under the broadest reasonable interpretation, the claimed first, second and third set of weights correspond to the local model parameters/weights exchanged among Zhu’s client nodes. Therefore, Zhu teaches or at least suggests the claimed weight exchange and average-weight generation. Applicant’s argument regarding Hardy is also not persuasive because Hardy is relied upon for the GAN aspect at each worker, while Zhu supplies the decentralized/no-central-server topology. The fact that Hardy may use server-side averaging in its own example does not negate the proposed combination. With respect to Lian and Mokhtari, Applicant’s argument is not persuasive because Lian is relied upon for the “decentralized parallel” gradient training aspect, not the full OSG algorithm or amended loss-function limitation. Lian teaches nodes running concurrently, communicating with neighbors, averaging received local variables with their own local variables, and updating using the average and local stochastic gradient without requiring gradient to be sent to a central node. Lian’s teaching correspond to Zhu’s teaching of decentralized. Mokhtari teaches the optimistic gradient/OGDA update technique for min-max or saddle point optimization. It would have been obvious to incorporate Mokhtari’s optimistic gradient update into the decentralized parallel framework of Zhu and Lian to perform known gradient-based optimization in a decentralized GAN training environment. Furthermore, Although Mokhtari’s analyze OGDA in a strongly convex-strongly concave setting, the rejection relies on Mokhtari only for the known optimistic-gradient update form, while Sanjabi is relied upon for the non-convex/non-concave min/max loss/objective function and the convergence property. Thus, the rejection does not require Mokhtari’s strongly convex-strongly concave objective to be the claimed non-convex/non-concave loss function. A person ordinary skill in the art would have been motivated to use the OGDA update technique to update the gradient, notwithstanding that Mokhtari’s convergence analysis is presented in the strongly convex-strongly concave setting, because such optimistic updates were known for min-max/saddle-point optimization and are useful for useful for improving stability while accounting for a wide range of parameter in adversarial gradient-based training. With respect to the amended loss function limitation, Applicant’s argument is also not persuasive. Sanjabi discloses an objective function for a min/max optimization problem, which under the broadest reasonable interpretation corresponds to the claimed loss function because it is the function being optimized. Sanjabi teaches that the objective function is maximized and minimized in a non-convex, non-concave min/max regime, wherein the Polyak- Lojasiewicz condition is imposed on the objective function structure of the non-convex, non-concave min/max problem so that the gradient algorithm can find a first-order stationary point within some iterations. Thus, Sanjabi teaches or at least suggests the claimed loss function comprising a non-convex, non-concave min/max function with provable non-asymptotic convergence to a first-order stationary point. Accordingly, the combine teachings of Zhu, Hardy, Lian, Mokhtari, and Sanjabi teach or at least suggest the amended claims. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1, 3-5, 10, 12-14, 19-20 are rejected under 35 U.S.C. 103 as being unpatentable in view of Zhu et.al (US 20220114475 A1), in view of Hardy et.al (NPL: MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets), further in view of Lian et.al (NPL: Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent), further in view of Mokhtari et.al (NPL: A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach), further in view of Sanjabi et.al (NPL: Solving Non-Convex Non-Concave Min-Max Games Under Polyak- Lojasiewicz Condition) Regarding claim 1, Zhu teaches or at least suggest a part of the 1 st limitation “ generating, by each of a plurality of nodes, gradients, wherein the gradients are based on a respective set of weights associated with each of the plurality of nodes of the neural network, ... ” (paragraph 41 “ When training the machine learning model related to the task, such as a deep neural network model, instead of sharing data, each client may share a respective update to the machine learning model computed at the client (e.g., in the form of local gradients) ”, and paragraph 43 “ First, each given client 22 obtains data randomly sampled from its local data ... Each given client 22 executes a machine learning algorithm to perform forward propagation ... to obtain a prediction for its local model 26, compute a loss for its local model 26 in accordance with a loss function, perform backpropagation to update its respective local model 26, and compute a gradient. The gradient may be represented as: Δw i = ∂f (w i ; ξ i ) / ∂ w i where w i denotes the set of parameters (or weights) of the local model 26 at the ith client 22 ”, and figure 2. Zhu discloses methods and systems for decentralized federated learning. Within the disclosure, Zhu discloses each local model compute a gradient based on the set of weights parameter, wherein each local model within the decentralized federated learning setting corresponds to a plurality of nodes of the neural network under the broadest reasonable interpretation, as claimed. Thus, Zhu teaches or at least suggests the generating of gradients based on weights by each nodes within the neural network as claimed.) Zhu teaches or at least suggest the 2 nd limitation “ performing a first round of exchanging weights between a first node and each of the plurality of nodes including a second node, including exchanging a first set of weights associated with the first node, and a second set of weights associated with a second node ” (paragraph 47 “ In federated learning, data sampled from a local dataset is not shared by a client. Instead, clients only exchange information about the parameter (e.g. weights) of their local models .”, and paragraph 70 “ FIG. 3 illustrates the information (e.g., local model parameters, and weighting coefficients) that may be communicated between clients 102 in a training round ”, and paragraph 68 “ the task owner client 110 may communicate the definition of the local model related to task (e.g., including the defined model parameters, defined model architecture, defined loss function, number of training rounds, etc.) to the participating clients 102 .”. Zhu discloses clients exchange information about the parameter (e.g. weights) of their local models with neighbor clients such as illustrated in figure 3. Within figure 3, client A exchange their weight information with client B and vice versa, wherein the client A and client B correspond to the first node and the second node within the claim respectively, and client A's weight and client B’s weight correspond to the first and second set of weight respectively. Furthermore, one of ordinary skilled in the art as suggested by the reference as the task owner client can communicate number of training rounds, thus the task owner can communicate a first round of weight exchange process, corresponding to the performing of weight exchange between a first and second node, as claimed.) Zhu teaches or at least suggest the 3 rd limitation “ performing a second round of exchanging weights between the second node and at least a third node, including exchanging the first set of weights received by the second node in the first round, with a third set of weights associated with the third node ” (figure 2, and paragraph 78 “ At 508, each given client 102 (e.g., the jth client) identifies a set of one or more neighbor client(s) 102 ... to which the given client 102 will transmit its respective set of local model parameters w j ... In some examples, step 508 is performed for each round of training ... and the given client 102 may use the set of identified neighbor client(s) 102 that was identified in an earlier training round ” and paragraph 80 “ all clients 102 may transmit information (i.e., respective set of local model parameters and weighting coefficient) to all other clients 102 ... Another possible implementation is for all clients 102 to be connected in a ring topology (e.g., a one-way ring or a two-way ring topology), in which case the number of transmissions is in proportion to the number of clients 102, which may help to reduce the communication overhead and help to achieve higher throughput of information between clients 102 ” Zhu discloses one of ordinary skilled in the art can communicate number of training rounds, thus a second round of weight exchange may be performed. Zhu further discloses a ring topology for weight exchange, such that all clients may transmit information (i.e., respective set of local model parameters) to all other clients, which correspond to the claimed process of exchanging the first set of weights received by the second node in the first round, with a third set of weights associated with the third node, wherein client C as illustrated in figure 2 and its information (weight) is corresponds to the third node with the third set of weight. Given the ring topology, repeated training rounds and the result of all clients may transmit information to all other clients, one of ordinary skilled in the art would inherently enables a second training round such that each client may forward their information that it previously received from another client in earlier training round to new client in subsequent rounds, such that all client received exchanged information from other clients, thus the weights of the client A can be exchanged via client B with the client C and client C can exchange its associated weights with client B. Since the claim does not limit the “ third set of weights associated with the third node ” must be the same or unmodified weights originally from the node, under BRI, the third set of weights reasonably corresponds to the current or updated local model parameter (weights) of the client C at the time of the second round of exchange.) Zhu teaches or at least suggest the 4 th limitation “ generating, for each of the plurality of nodes, an average weight based at least in part on the first set of weights, the second set of weights, and the third set of weights ” (paragraph 88 “ At 514, each given client 102 aggregates the set(s) of local model parameters received at 512. For example, the set(s) of local model parameters may be aggregated by computing a weighted sum or weighted average ” Zhu discloses each set of local models at each client may aggregate the sets of parameters received from other clients and compute a weighted average, wherein the sets of parameter (weights) being exchanged with other clients may comprise of weights from client A, B and C, which teach or at least suggest the generating an average weight, as claimed.) Zhu teaches or at least suggest a part of the 5 th limitation “ reducing incidence of a bottleneck in network traffic from being formed at a central node when training the neural network, by updating the first set of weights, the second set of weights, and the third set of weights ... based on the gradients and the average weight for each of the plurality of nodes... a decentralized ... algorithm in which the gradients need not be sent to any central node ... ” (paragraph 102 “ the scheduler 202 may manage scheduling of data communication among clients 102 to avoid bottlenecks in the communication resources or congestion in the system 600 ”, paragraph 128 “ The disclosed examples enable privacy of data at a client to be maintained during decentralized federated learning of a local model related to a task, because data from local datasets do not need to be shared outside of the client ... Because clients communicate mainly directly with each other during training, there may be lower communication latency and less data network traffic compared to other existing techniques, which may lead to faster response time and shorter request queues ...”, paragraph 91 “ The given client 102 updates its respective set of local model parameters ... using the computed aggregations .”, and paragraph 92 “ The set of local model parameters w j is then updated based on the gradient vector ” Zhu discloses the decentralized federated learning scheme, which can be managed by a scheduler that manage data communication among clients to avoid bottlenecks, wherein the scheduler is adapted in a decentralized federated learning setting that each client communicates mainly directly with each other during training and does not require a single central node, thus there may be lower communication latency and less data network traffic. Zhu also discloses that after each client communicate their information with other clients, each client may update their own model parameters (weights) based on the computed average and gradient vector, which corresponds to the updating at each of the plurality of nodes in the decentralized setting, as claimed.) Zhu does not teach a part of the 1 st limitation “ ... wherein the neural network is part of a generative adversarial network ”. However, Hardy teaches or at least suggest this part of the limitation (page 3 section III-c) “ Federated learning ... proposes to train a machine learning model, and in particular a deep neural network, on a set of workers ... we propose an adapted version of federated learning to GANs. This adaptation considers the discriminator D and generator G on each worker as one computational object to be treated atomically. Workers perform iterations locally on their data and every E epochs (i.e., each worker passes E times the data in their GAN) ” Hardy discloses adaptation of federated learning to GANs, in which each worker may comprises of a discriminator D and generator G that each performed their training locally. The GANs represented the generative adversarial network within the claim, and the discriminator and the generator represent the neural network as part of the generative adversarial network within the claim.) Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of methods and systems for decentralized federated learning comprising of each client exchange information with others by Zhu, with the teaching of federated learning GAN by Hardy. The motivation to do so is referred to in Hardy’s disclosure (page 3 section III-c) “ Federated learning ... proposes to train a machine learning model, and in particular a deep neural network, on a set of workers. It follows the parameter server framework, with the particularity that workers perform numerous local iterations between each communication to the server (i.e., a round) ... This adaptation considers the discriminator D and generator G on each worker as one computational object to be treated atomically ” Hardy discloses a possibility of an adaptation of GAN on federated learning scheme, in which each worker comprises of the discriminator D and generator G on their own to perform local learning. Given that Zhu also discloses a federated learning process based on decentralized setting, a plurality of clients communicates with each other, and Zhu discloses at paragraph 131 “ The examples disclosed herein may be used for learning various machine learning models ..., and other neural network architectures ”, which suggest that the decentralized federated learning paradigm may be applied for any machine learning models and neural network architectures. One of ordinary skilled in the art would have been motivated to combine the GAN model by Hardy into the decentralized federated learning setting by Zhu, such that each worker in Hardy correspond to each client in Zhu, and the GAN model of each worker can be improved by training with the decentralized federal learning setting.) Zhu/Hardy does not teach part of the 5 th limitation “ ... a decentralized parallel ... algorithm in which the gradients need not be sent to any central node, ... ”. However, Lian teaches or at least suggest this limitation (Page 2, Section 1 “ The potential bottleneck of the centralized network topology lies on the communication traffic jam on the central node, because all nodes need to communicate with it concurrently iteratively. The performance will be significantly degraded when the network bandwidth is low. These motivate us to study algorithms for decentralized topologies, where all nodes can only communicate with its neighbors and there is no such a central node, shown in Figure 1(b) ”, Page 6 section 3 Algorithm 1 “ The D-PSGD algorithm is a synchronous parallel algorithm. All nodes are usually synchronized by a clock. Each node maintains its own local variable and runs the protocol in Algorithm 1 concurrently ”, and Page 12, Section 5.3 “ D-PSGD allows more communication in an efficient way without reaching the network bottleneck ”. Lian discloses the incidence of bottleneck within the network traffic occur over the communication traffic jam on the central node can be overcome by applying the parallel stochastic gradient descent algorithm on decentralized network topologies. The decentralized parallel learning setting provides communication of nodes only with its neighbors and there is no central node, which help reducing incidence of bottleneck in network traffic from being formed as the gradients need not be sent to any central node but from one node to another node, since there is no central node in the decentralized network topology and the learning can be between each node can be performed in parallel. The teaching of an OSG is incorporated based on another reference incorporated in below paragraphs.) Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of methods and systems for decentralized federated learning comprising of each client exchange information with others by Zhu, and the teaching of federated learning GAN by Hardy, with the teaching of a decentralized parallel stochastic algorithm by Lian. The motivation to do so is referred to in Lian’s disclosure (Page 12, Section 5.3 “ D-PSGD allows more communication in an efficient way without reaching the network bottleneck ”, Lian discloses a Decentralized parallel stochastic gradient descent algorithm (D-PSGD) that does not specify any central node; thus, no gradients need to be sent to central node and the learning may be performed in parallel. Zhu similarly discloses a decentralized learning setting among client without relying on a central node or server, but does not teach the learning as being occur in parallel. Therefore, the teaching of Zhu can further incorporate the teaching by Lian based on the similarity in the decentralized learning setting and the incorporation of the parallel learning between each node (client) to further improve the training paradigm and reduce the training time.) Zhu/Hardy/Lian does not teach a part of the 5 th limitation “ ... an optimistic stochastic gradient (OSG) algorithm ..., wherein the OSG algorithm is configured to optimize a loss function for the neural network, ..., the OSG algorithm comprising a plurality of update operations based on different gradient types ”. However, Mokhtari teaches or at least suggest this part of the limitation (Page 1 Section 1 equation (1) “ Motivated by the interest in computational methods for solving the minmax problem in (1), in this paper we consider convergence rate analysis of discrete-time gradient based optimization algorithms for finding a saddle point of problem (1). We focus on ... Optimistic Gradient Descent Ascent (OGDA) method ”, Page 5 section 4.1 algorithm 1 “ OGDA method for saddle point problems ...1: for k = 1, 2, . . . do ... ”, and Page 7 section 4.2 “ This implies that in the OGDA update at step k, the coefficient of the current gradient, ... should be exactly twice the coefficient of the negative of the previous gradient ”. Mokhtari discloses the application of Optimistic Gradient Descent/Ascent (OGDA) toward min-max optimization problems of equation (1), wherein the OGDA is one of the focus algorithm to provide optimization toward the objective function f(x,y), which represents a loss function of the GAN neural network for a saddle point optimization problem. A current gradient and a previous gradient are calculated to determine the update for the next state, which corresponds to different gradient types within the claim, and there is more than one update operation as represented by the algorithm which recites “ 1: for k = 1, 2, ... do ... ”, which corresponds to a plurality of update operations. In other words, at each k update operation (line 1 within the algorithm 1), the OGDA algorithm compute a current gradient and the previous gradient (line 2 and 3 within the algorithm 1) to determine the update operation.) Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of methods and systems for decentralized federated learning comprising of each client exchange information with others by Zhu, and the teaching of federated learning GAN by Hardy, and the teaching of a decentralized parallel stochastic algorithm by Lian, with the teaching of analysis of Extra-gradient and Optimistic Gradient methods to solve saddle point problems by Mokhtari. The motivation to do so is referred to in Mokhtari’s disclosure (Page 10, section 6 “ As we can see, EG and OGDA perform better than GDA and their convergence paths are closer to the one for PP which has the fastest rate. This observation matches our theoretical claim that EG and OGDA are more accurate approximations of PP relative to GDA ”. Mokhtari discloses through the analysis that the OGDA perform better than GDA in solving saddle point problems, as the OGDA provides more accurate approximations of proximal point (PP) relative to GDA. While the teaching combination of Zhu/Hardy/Lian only provides stochastic gradient algorithms, the teaching combination can further incorporate the teaching by Mokhtari to incorporate the algorithm of OGDA to calculate gradients that consider past and current gradient to update the model accordingly for improvement in future performance.) Zhu/Hardy/Lian/Mokhtari does not teach a part of the limitation “... wherein the loss function comprises a non-concave, non-convex min/max function with provable non-asymptotic convergence to a first-order stationary point ”. However, Sanjabi teaches this part of the limitation (page 1 section 1 “ This optimization problem can be viewed as a zero-sum game between two players where the goal of the first player is to maximize the objective function f(·, ·), while the other player’s objective is to minimize the objective function. Our goal is to develop an algorithm for finding a first order stationary point of the above optimization problem in the non-convex non-concave regime .”, and page 4 section 3.2.1 “ in order to obtain an ε– stationary solution of the game (1), O(ε −2 ) evaluations of the gradient of the objective with respect to θ is needed ”. Sanjabi disclose an objective function for a min/max optimization problem with minimizing and maximizing the objective function, which under the broadest reasonable interpretation corresponds to the claimed loss function. Sanjabi teaches solving the non-convex, non-concave min/max optimization problem corresponding to the GAN model under the Polyak-Lojasiewicz condition. The Polyak-Lojasiewicz condition is imposed on the objective function of the non-convex/non-concave min/max problem to permit provable convergence of the gradient descent/ascent algorithm to an ε first order stationary point within O(ε −2 ) iterations, thereby also demonstrating the non-asymptomatic convergence behavior, since there is a finite number of iteration instead of infinite.) Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of methods and systems for decentralized federated learning comprising of each client exchange information with others by Zhu, the teaching of federated learning GAN by Hardy, the teaching of a decentralized parallel stochastic algorithm by Lian, and the teaching of analysis of Extra-gradient and Optimistic Gradient methods to solve saddle point problems by Mokhtari, with the teaching of finding the first order stationary points on the non-convex nonconcave min-max problem under Polyak- Lojasiewicz (PL) Condition by Sanjabi. The motivation to do so is referred to in Sanjabi’s disclosure (page 1 section abstract “ we consider the problem of solving a min-max zero sum game. This problem has been extensively studied in the convex concave regime where the global solution can be computed efficiently ... we show that a simple multi-step gradient descent-ascent algorithm finds an ε–first order stationary point of the problem in O(ε −2 ) iterations ”, and page 1 section 1 “ Our goal is to develop an algorithm for finding a first order stationary point of the above optimization problem in the non-convex non-concave regime ” Sanjabi discloses for the min-max loss function corresponding to GAN training, applying a gradient descent-ascent under the Polyak- Lojasiewicz Condition leads to convergence to a first-order stationary point within O(ε −2 ) iterations. While the teaching combination recites a decentralized multi-client setting with each client may employs a GAN model, the teaching combination does not disclose the necessary number of training rounds and solving the min-max problem in the non-concave non-convex regime through an optimized function. However, Sanjabi discloses this objective function and the number of iterations given by the Polyak- Lojasiewicz (PL) Condition appliedto the objective function to obtain convergence on the first stationary point on the non-convex non-concave min-max regime. Therefore, one of ordinary skilled in the art would have been motivated to combine the teaching combination with the teaching of Sanjabi in order to apply the objection function with the condition to solve the non-convex non-concave min-max problem of the GAN model while perform the necessary number of iterations that help find the first order stationary point.) Regarding claim 3 depends on claim 1, thus the rejection of claim 1 is incorporated. Zhu teaches or at least suggests the limitation “ the weights are exchanged for a predetermined number of rounds of exchanges that ranges from 2 -10 rounds of exchanges ” (paragraph 80 and figure 2 “ Various techniques may be used at a given client 102 to identify the neighbor clients(s) for transmitting information ... Another possible implementation is for all clients 102 to be connected in a ring topology (e.g., a one-way ring or a two-way ring topology), in which case the number of transmissions is in proportion to the number of clients 102 ”, Zhu discloses a possible implementation is for all clients to be connected in a ring topology, such that the number of information transmissions is in proportion to the number of clients. Given the illustration in example figure 2, one ordinary skilled in the art would recognize that the figure illustrates a ring topology with the number of clients up to 4, thus there can be 4 rounds of information transmission at each training round, which is analogous to the number of rounds of exchanges that ranges from 2 -10 rounds within the claim.) Regarding claim 4 depends on claim 1, thus the rejection of claim 1 is incorporated. Zhu teaches or at least suggests the limitation “ generating the average weight comprises calculating the average weight over each exchange of a predetermined amount of exchanges that the weights are exchanged for ” (paragraph 88 “ At 514, each given client 102 aggregates the set(s) of local model parameters received at 512. For example, the set(s) of local model parameters may be aggregated by computing a weighted sum or weighted average ”. Zhu discloses the local model may compute a weighted average based on the sets of local model parameters exchanged from other clients which is analogous to the claimed calculating the average weight over each exchange of weights within the claim.) Regarding claim 5 depends on claim 1, thus the rejection of claim 1 is incorporated Zhu teaches or at least suggests the limitation “ a first type of update operation that updates the first set of weights, the second set of weights, and the third set of weights based on gradients from a previous weight update iteration, to yield resulting set of weights, wherein the previous weight update iteration comprises a weight update iteration that is previous to a present weight update iteration ”. (paragraph 91 “ The given client 102 updates its respective set of local model parameters and its respective weighting coefficient using the computed aggregations ... such as adding the aggregation to the existing set of local model parameters ”, and paragraph 92 “ The set of local model parameters w j is then updated based on the gradient vector ”, and paragraph 93 “ If the convergence condition is not satisfied, the method 500 returns to step 508 to perform another round of training .” Zhu discloses multiple rounds of training, including each client exchange local model parameter (weights) with another client, and each client may then receive the exchange parameter (weights) and perform their own local model parameter update based on the computed average parameters (weights), and gradient vector, wherein the gradient vector may be calculated from a previous training round as configured by one of ordinary skilled in the art when applying the teaching of OGDA by Mokhtari to calculate the previous gradient to update the model.) Zhu teaches or at least suggests the limitation “ a second update operation that further updates the resulting sets of weights based on gradients from the present weight update iteration ” (paragraph 91 “ The given client 102 updates its respective set of local model parameters and its respective weighting coefficient using the computed aggregations ... such as adding the aggregation to the existing set of local model parameters ”, and paragraph 92 “ The set of local model parameters w j is then updated based on the gradient vector ”, and paragraph 93 “ If the convergence condition is not satisfied, the method 500 returns to step 508 to perform another round of training .” Zhu discloses the repeat of another training round until convergence condition is satisfied, in which each client may further transmit each local model parameter (weights) with each other and further update their own weights using gradients that can be calculated at the current training round as configured by one of ordinary skilled in the art when applying the teaching of OGDA by Mokhtari to calculate the gradient of the current training round to update the model.) Regarding claim 10, the applicant is further directed to the rejections of claim 1 as set forth above, as claim 10 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 12 depends on claim 10, thus the rejection of claim 10 is incorporated. The applicant is further directed to the rejections of claim 3 as set forth above, as claim 12 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 13 depends on claim 10, thus the rejection of claim 10 is incorporated. The applicant is further directed to the rejections of claim 4 as set forth above, as claim 13 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 14 depends on claim 10, thus the rejection of claim 10 is incorporated. The applicant is further directed to the rejections of claim 5 as set forth above, as claim 13 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 19, Zhu teaches or at least suggests the limitation “ a computer readable storage medium including computer program code that, when executed on one or more computer processors ” (paragraph 66 “ In some example embodiments, the memory(ies) 128 may include software instructions for execution by the processing device 114 to implement a module for decentralized federated learning 200, as discussed further below ... provided executable instructions by a transitory or non-transitory computer-readable medium ”, and paragraph 135 “ The machine-executable instructions may be in the form of code sequences, configuration information, or other data, which, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in a method according to example embodiments of the present disclosure .”. Zhu discloses memories may include software instructions for execution by the processing device, in which the software instructions may be executed via a processor and the memories may be a transitory or non-transitory computer-readable medium.) The applicant is further directed to the rejections of claim 1 as set forth above, as claim 19 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 20 depends on claim 19, thus the rejection of claim 19 is incorporated. The applicant is further directed to the rejections of claim 5 as set forth above, as claim 20 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 21 depends on claim 20, thus the rejection of claim 20 is incorporated. Zhu in view of Sanjabi teaches or at least suggest the limitation “ wherein the number of rounds of exchanges is determined to be of a sufficient size for average approximation for a centralized network topology such that localized averaging at each node converges a proposed solution of non-concave, non-convex, min/max function with provable non-asymptomatic convergence to a first-order stationary point, the min/max function comprising the loss function, wherein the proposed solution being converged is determined by a given round of the number of rounds of exchanges updating the proposed solution by no more than an insignificance threshold; ” (paragraph 93 “ At 518, a determination is made whether a predefined convergence condition (e.g., any suitable convergence condition as discussed previously) has been satisfied. The determination whether the convergence condition is satisfied may be performed by the task owner client 110 ”, and paragraph 94 “ The convergence condition may be defined by the task owner client 110, as part of defining the local model related to the task T at step 502. For example, the convergence condition may include one or more of: that the number of training rounds meets or exceeds a defined maximum number ”. Zhu discloses the predefined convergence condition made by one of ordinary skilled in the art such as by determining the convergence when the number of training rounds meets or exceeds a defined maximum number. Given that each client within the decentralized multi-party learning setting, wherein each client employs their own GAN model as disclosed by the teaching combination above, one of ordinary skilled in the art would recognize the goal of GAN model learning is to utilize the average weights to obtain convergence on a stationary point of the min/max function of each GAN model, thus on one of ordinary skilled in the art may determine the predefined number of training rounds to lead the GAN model to such convergence. Sanjabi then teaches solving the non-convex, non-concave min/max optimization problem corresponding to the GAN model under the Polyak-Lojasiewicz condition. The Polyak-Lojasiewicz condition is imposed on the objective function of the non-convex/non-concave min/max problem to permit provable convergence of the gradient descent/ascent algorithm to an ε first order stationary point within O(ε −2 ) iterations, thereby also demonstrating the non-asymptomatic convergence behavior, since there is a finite number of iteration instead of infinite.) Zhu teaches or at least suggest the limitation “ wherein the average approximation is provided despite each node determining a localized average, ... ” (paragraph 88 “ At 514, each given client 102 aggregates the set(s) of local model parameters received at 512. For example, the set(s) of local model parameters may be aggregated by computing a weighted sum or weighted average .” Zhu discloses the computing of an average computation for the local model parameters based on parameters received from other clients. Accordingly, one of ordinary skilled in the art may perform Zhu’s average computation as an average approximation based on the predetermined number of training rounds and number of clients participating in each round before each client execute their exchanges and update, such that this average computation may serve as a baseline of average computation (in place of the centralized learning setting since the current teaching combination is decentralized) for all clients to follow in term of how close the calculated average parameters of each client at each training round is, such that convergence of model training across all clients may be obtained.) The applicant is further directed to the rejections of claim 1 as set forth above, as claim 21 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 22 depends on claim 21, thus the rejection of claim 21 is incorporated. The applicant is further directed to the rejections of claim 5-9 as set forth above, as claim 22 is similarly rejected based on the same rationale, since claim 22 recites similar limitations and processing steps to set of claims 5-9. Claims 6-9, 15-18, 22 are rejected under 35 U.S.C. 103 as being unpatentable in view of Zhu et.al (US 20220114475 A1), in view of Hardy et.al (NPL: MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets), further in view of Lian et.al (NPL: Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent), further in view of Mokhtari et.al (NPL: A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach), further in view of Sanjabi et.al (NPL: Solving Non-Convex Non-Concave Min-Max Games Under Polyak-Lojasiewicz Condition), further in view of Voinea et.al (US 11797705 B1). Regarding claim 6 depends on claim 5, thus the rejection of claim 5 is incorporated. Hardy teaches or at least suggests a part of the limitation “ The method of claim 5, wherein the neural network is a generator having generator weights, ... ” (page 3 section III-c) “ FL-GAN ... By the design of GANs, a generator and a discriminator are two separate elements that are yet tightly coupled; ... This adaptation considers the discriminator D and generator G on each worker as one computational object to be treated atomically ”, and page 4 section IV table 1 figure 1b “ FL-GAN uses generators ... on each worker ... w (resp. θ) Parameters of G (resp. D) ” Hardy discloses each worker comprise of each generator and discriminator as illustrated in figure 1b, in which each worker may correspond to each client within the teaching by Zhu and each generator and discriminator within each worker comprise of parameters w, wherein the parameters may be weight parameter as taught by Zhu. In other word, the machine learning model of each worker (client) within the decentralized federated learning setting by Zhu may be a GAN model having the generator and its corresponding weights parameter.) Zhu/Hardy/Lian/Mokhtari does not teach the limitation “ ... wherein the first type of update operation comprises updating a first set of the generator weights (U LGW1 ) as a function of an average generator weight (A GW ), a learning rate (L), and gradients based on generator weights from the previous weight update iteration (G GW_PREV ), such that U LGW1 = A GW - L * G GW_ PREV , wherein G GW_ PREV is determined based on U LGW1 from the previous weight update iteration ” within the limitation. However, Voinea teaches or at least suggests this limitation (Column 10 lines 44-50, where Voinea discloses “ Based on respective losses, generator 505 and discriminator 510 may update their respective parameters (e.g., weights and biases), for instance, with backpropagation and descent gradients. For example, (a new value of weight) = (an old value of the weight) - (learning rate) × (partial gradient of the loss with respect to the weight) ”. Voinea discloses an update equation to update weight of generator and discriminator within the GAN network. This update equation is an example of using gradient descent with subtraction to find minimums which is analogous to the update equation for generator weight within the claim, wherein one of ordinary skilled in the art may configure to modify this function in accordance with the teaching by Zhu which recites the computed average weights along with multiple training rounds and the teaching by Mokhtari which recites calculating the previous gradient of the OGDA method at page 7 section 4.2, with the motivation to combine the teachings is recited below.) Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teaching of methods and systems for decentralized federated learning comprising of each client exchange information with others by Zhu, and the teaching of federated learning GAN by Hardy, and the teaching of a decentralized parallel stochastic algorithm by Lian, and the teaching of analysis of Extra-gradient and Optimistic Gradient methods to solve saddle point problems by Mokhtari with the teaching of using descent gradients and weight update equation on GAN network to obtain weights for generator and discriminator in GAN by Voinea. The motivation to do so is referred to in Voinea’s disclosure (Column 10 lines 44-50, where Voinea discloses “ (a new value of weight) = (an old value of the weight) - (learning rate) × (partial gradient of the loss with respect to the weight) ”. Voinea discloses a conventional method to update weight given previous weight value. While the teaching by Zhu discloses update the weight using gradients and average weights, Zhu does not disclose a specific calculation to perform such update. Therefore, one of ordinary skilled in the art can incorporate the weight update calculation equation to help calculate the update weights within the teaching by Zhu and provide an accurate result of the updated weights. The calculating of weights with gradient can be further improved by applying the OGDA method by Mokhtari (with the motivation to combine as explained above) to account for the optimistic characteristic which is represented by the current and previous gradients within the method, since Zhu also discloses multiple training rounds until convergence.) Regarding claim 7 depends on claim 6, thus the rejection of claim 6 is incorporated. Voinea teaches or at least suggests the limitation “ the second update operation comprises updating a second set of the generator weights (U LGW2 ) as a function of an average generator weight (A GW ), a learning rate (L), and gradients based on generator weights from the present weight update iteration (G GW_PRESENT ), such that U LGW2 = A GW - L * G GW_ PRESENT , wherein G GW_ PRESENT is determined based on U LGW1 from the present weight update iteration ” (Column 10 lines 44-50, where Voinea discloses “ Based on respective losses, generator 505 and discriminator 510 may update their respective parameters (e.g., weights and biases), for instance, with backpropagation and descent gradients. For example, (a new value of weight) = (an old value of the weight) - (learning rate) × (partial gradient of the loss with respect to the weight) ”. Voinea discloses an update equation to update weight of generator and discriminator within the GAN network. This update equation is an example of using gradient descent with subtraction to find minimums which is analogous to the update equation for generator weight within the claim, wherein one of ordinary skilled in the art may configure to modify this function in accordance with the teaching by Zhu which recites the computed average weights along with multiple training rounds and the teaching by Mokhtari which recites calculating the current gradient of the OGDA method at page 7 section 4.2.) Regarding claim 8 depends on claim 5, thus the rejection of claim 5 is incorporated. Hardy teaches or at least suggests a part of the limitation teaches a part of the limitation “ the method of claim 5, wherein the neural network is a discriminator having discriminator weights, ... ” (page 3 section III-c) “ FL-GAN ... By the design of GANs, a generator and a discriminator are two separate elements that are yet tightly coupled; ... This adaptation considers the discriminator D and generator G on each worker as one computational object to be treated atomically ”, and page 4 section IV table 1 figure 1b “ FL-GAN uses generators ... on each worker ... w (resp. θ) Parameters of G (resp. D) ” Hardy discloses each worker comprise of each generator and discriminator as illustrated in figure 1b, in which each worker may correspond to each client within the teaching by Zhu and each generator and discriminator within each worker comprise of parameters w, wherein the parameters may be weight parameter as taught by Zhu. In other word, the machine learning model of each worker (client) within the decentralized federated learning setting by Zhu may be a GAN model having the discriminator and its corresponding weights parameter.) Zhu/Hardy/Lian/Mokhtari/Sanjabi does not teach the limitation “ ... wherein the first update operation comprises updating a first set of discriminator weights (U LDW1 ) as a function of an average discriminator weight (A DW ), a learning rate (L), and gradients based on discriminator weights from the previous weight update iteration (G DW_PREV ), such that U LDW1 = A DW + L * G DW_ PREV , wherein G GW_ PREV is determined based on U LDW1 from the previous weight update iteration ” However, Voinea teaches or at least suggests this limitation (Column 10 lines 44-50, where Voinea discloses “ Based on respective losses, generator 505 and discriminator 510 may update their respective parameters (e.g., weights and biases), for instance, with backpropagation and descent gradients. For example, (a new value of weight) = (an old value of the weight) - (learning rate) × (partial gradient of the loss with respect to the weight) ”. Voinea discloses an update equation to update weight of generator and discriminator within the GAN network. This update equation is an example of using gradient descent with subtraction to find minimums, wherein one of ordinary skilled in the art may configure to modify this function as an addition process to account for the maximization in evaluating the discriminator’s result. The equation can be further modified in accordance with the teaching by Zhu which recites the computed average weights along with multiple training rounds and the teaching by Mokhtari which recites calculating the previous gradient of the OGDA method at page 7 section 4.2, with the motivation to combine the teachings is similar to the motivation in claim 6.) The motivation to combine the teaching of Zhu/Hardy/Lian/Mokhtari/Sanjabi with the teaching of Mokhtari is similar to the motivation as recited in claim 6, since both claims recites similar updating procedure to obtain an updated weight based on gradient from a previous weight update round. Regarding claim 9 depends on claim 8, thus the rejection of claim 8 is incorporated. Voinea teaches or at least suggests the limitation “ the second type of update operation comprises updating a second set of the discriminator weights (U LDW2 ) as a function of an average discriminator weight (A DW ), a learning rate (L), and gradients based on discriminator weights from the present weight update iteration (G DW_PRESENT ), such that U LDW2 = A DW + L * G DW_ PRESENT , wherein G DW_ PRESENT is determined based on U LDW1 from the present weight update iteration ” (Column 10 lines 44-50 where Voinea discloses “ Based on respective losses, generator 505 and discriminator 510 may update their respective parameters (e.g., weights and biases), for instance, with backpropagation and descent gradients. For example, (a new value of weight) = (an old value of the weight) - (learning rate) × (partial gradient of the loss with respect to the weight) ”. Voinea discloses an update equation to update weight of generator and discriminator within the GAN network. This update equation is an example of using gradient descent with subtraction to find minimums, wherein one of ordinary skilled in the art may configure to modify this function as an addition process to account for the maximization in evaluating the discriminator’s result. The equation can be further modified in accordance with the teaching by Zhu which recites the computed average weights along with multiple training rounds and the teaching by Mokhtari which recites calculating the current gradient of the OGDA method at page 7 section 4.2.) Regarding claim 15 depends on claim 14, thus the rejection of claim 14 is incorporated. The applicant is further directed to the rejections of claim 6 as set forth above, as claim 15 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 16 depends on claim 15, thus the rejection of claim 15 is incorporated. The applicant is further directed to the rejections of claim 7 as set forth above, as claim 16 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 17 depends on claim 14, thus the rejection of claim 14 is incorporated. The applicant is further directed to the rejections of claim 8 as set forth above, as claim 17 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Regarding claim 18 depends on claim 17, thus the rejection of claim 14 is incorporated. The applicant is further directed to the rejections of claim 9 as set forth above, as claim 18 is similarly rejected based on the same rationale, since both claims recites similar limitations and processing steps. Conclusion 07-40 AIA Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL . See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUY TU DIEP whose telephone number is (703)756-1738. The examiner can normally be reached M-F 8-4:30. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /DUY T DIEP/Examiner, Art Unit 2123 /ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123 Application/Control Number: 17/081,779 Page 2 Art Unit: 2123 Application/Control Number: 17/081,779 Page 3 Art Unit: 2123 Application/Control Number: 17/081,779 Page 4 Art Unit: 2123 Application/Control Number: 17/081,779 Page 5 Art Unit: 2123 Application/Control Number: 17/081,779 Page 6 Art Unit: 2123 Application/Control Number: 17/081,779 Page 7 Art Unit: 2123 Application/Control Number: 17/081,779 Page 8 Art Unit: 2123 Application/Control Number: 17/081,779 Page 9 Art Unit: 2123 Application/Control Number: 17/081,779 Page 10 Art Unit: 2123 Application/Control Number: 17/081,779 Page 11 Art Unit: 2123 Application/Control Number: 17/081,779 Page 12 Art Unit: 2123 Application/Control Number: 17/081,779 Page 13 Art Unit: 2123 Application/Control Number: 17/081,779 Page 14 Art Unit: 2123 Application/Control Number: 17/081,779 Page 15 Art Unit: 2123 Application/Control Number: 17/081,779 Page 16 Art Unit: 2123 Application/Control Number: 17/081,779 Page 17 Art Unit: 2123 Application/Control Number: 17/081,779 Page 18 Art Unit: 2123 Application/Control Number: 17/081,779 Page 19 Art Unit: 2123 Application/Control Number: 17/081,779 Page 20 Art Unit: 2123 Application/Control Number: 17/081,779 Page 21 Art Unit: 2123 Application/Control Number: 17/081,779 Page 22 Art Unit: 2123 Application/Control Number: 17/081,779 Page 23 Art Unit: 2123 Application/Control Number: 17/081,779 Page 24 Art Unit: 2123 Application/Control Number: 17/081,779 Page 25 Art Unit: 2123 Application/Control Number: 17/081,779 Page 26 Art Unit: 2123 Application/Control Number: 17/081,779 Page 27 Art Unit: 2123
Prosecution Insights