Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/2/2026 has been entered.
Remarks
This Office Action is responsive to Applicants' Amendment filed on March 2, 2026, in which claims 1-2, 4, 6-7, 9-11, 13, 15, and 17-20 are currently amended. Claims 1-20 are currently pending.
Response to Arguments
The rejections to claims 1-9 under 35 U.S.C. § 101 are hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1-20 under 35 U.S.C. 103 based on amendment have been considered and are persuasive. The argument is moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4, 7, 8, 10, 11, 13, 16-20 are rejected under U.S.C. §103 as being unpatentable over the combination of Huang (US11586911B2) and Tucker (“Inverse reinforcement learning for video games”, 2018).
Regarding claim 1, Huang teaches A system for deep reinforcement learning, the system comprising: a processor; ([Abstract] "A pre-training apparatus and method for reinforcement learning based on a Generative Adversarial Network (GAN) is provided" [Col. 6 l. 17-33] "Apparatus 12 includes processing circuitry 18. Processing circuitry 18 includes processor 20 and memory 22. In addition to a traditional processor and memory, processing circuitry 18 may comprise integrated circuitry for processing and/or control, e.g., one or more processors and/or processor cores ")
a first neural network implemented on the processor; ([Col. 8 l. 16-20] "System 10 includes GAN 28 that includes generator 30" [Col. 6 l. 43-47] "pre-training code 24 includes instructions that, when executed by processor 20, causes processor 20 to perform the functions described herein such as the functions described with respect to FIGS. 4 and 5")
a second neural network implemented on the processor, the second neural network being different from the first neural network;([Col. 8 l. 16-20] "System 10 includes GAN 28 that includes generator 30 and discriminator 32")
and a memory storing instructions that, when executed by the processor, cause the processor to:([Col. 6 l. 42-46] "memory 22 is configured to store pretraining code 24. For example, pre-training code 24 includes instructions that, when executed by processor 20, causes processor 20 to perform the functions described herein such as the functions described with respect to FIGS. 4 and 5")
generate, by the first neural network, a synthetic data based on an original data, ([Col. 9 l. 57-61] "Processing circuitry 18 is configured to cause generator 30 trained with training data to generate first synthetic data (Block S122). In one or more embodiments, generator 30 trained with minibatches of data D1 (s, a, s', r) generates a batch of data D2(s, a, s', r).")
provide the original data and the generated synthetic data to the second neural network, ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 9 l. 20-50] "Merge Sl (s,a) and S2 (s',r) to form a batch of D3 (s,a,s'r) Feed D3 (s,a,s'r) as real data into GAN for a training session [...] train generator 30 and discriminator 32 using the training data (Block S120). In one or more embodiments, generator 30 and discriminator 32 are trained with minibatches or portions of training data, e.g., D1 (s, a, s', r)" Huang explicitly trains the discriminator (second network) with original data D1, and feeds synthetic data S2 along with input data S1)
wherein the first neural network and the second neural network are structured to continuously learn using deep reinforcement learning including a state, an action, a reward, and a next state, ([Col. 9 l. 20-45] "Repeat […] Until GAN converges [...] the pre-training procedure can be updated [...] the training data includes state (s), action (a), transitioned to state (s′) for choosing action (a), and reward (r) for choosing action (a) such that training data is written as D(s, a, s′, r), e.g., D1(s, a, s′, r)" Huang explicitly discloses all four reinforcement learning fields and Algorithm 2 repeatedly trains the GAN until convergence or timeout with continuous update when later samples become available)
wherein the state is a value of an input feature in the original data, ([Col. 9 l. 40] "the training data includes state (s)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" In Huang the state s is literally an input field/value in the original data)
the action [is continuous and] includes the synthetic data, ([Col. 9 l. 40] "the training data includes […] action (a)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" Huang discloses that synthetic data D2/D3/D4 include action values a as part of the generated tuple)
the reward is a measure of how unsuccessful the second neural network is in discriminating the original data from the synthetic data, ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly uses the reward value as a measure to determine the loss, the loss is in turned used to determine convergence, the convergence explicitly being when the discriminator reaches a point of being unsuccessful in discriminating the original data from the synthetic data)
and the next state is a next group of examples to generate a next iteration of the synthetic data based on the original data,([Col. 9 l. 35-45] "Generate a batch of D2 (s,a,s′,r) via the generator G […] Get as output a batch of S2 (s′,r) from DNN Merge S1 (s,a) and S2 (s′,r) to form a batch of D3 (s,a,s′r) […] transitioned to state (s')" Huang describes the next state s' as part of the RL tuple and discloses successive generated batches D2, D3, D4 relying on s' across iterations)
generate, by the second neural network a prediction identifying the original data and the generated synthetic data, and ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator")
based at least in part on the prediction incorrectly identifying the generated synthetic data, export the generated synthetic data. ([Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly trains the GAN until it converges by successfully tricking the discriminator into incorrectly identifying enough of the synthetic data, at which point the synthetic data is output/exported for experience replay data).
However, Huang does not explicitly teach the action is continuous.
Tucker, in the same field of endeavor, teaches the action is continuous([p. 3 §3] "We build on adversarial IRL (AIRL), an algorithm achieving state-of-the-art performance on simulated robotics tasks (Fu et al., 2017). The reference implementation of AIRL assumes a continuous action space and uses a fully-connected policy and reward network [...] Adversarial IRL formulates the inverse reinforcement learning problem as a GAN (Goodfellow et al., 2014). We learn a reward function fθ(s,a) for taking action a in state s and a stochastic policy π(a | s). The policy is the generator, and is trained using forward RL on the reward function fθ(s,a). The discriminator is restricted to have the special form").
Huang as well as Tucker are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, Huang as well as Tucker are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Huang with the teachings of Tucker by using a continuous action space. One of ordinary skill in the art would realize that the action space must be either discrete or continuous and Tucker explicitly acknowledges continuous action space in reinforcement learning generative adversarial networks and provides as additional motivation for combination ([p. 1 §1] “Recent deep IRL algorithms have achieved good performance on a variety of continuous control tasks”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Huang and Tucker teaches The system of claim 1, wherein the instructions further cause the processor to: based at least in part on the prediction correctly identifying the generated synthetic data, control the first neural network to execute a machine learning model to calculate a loss for the generated synthetic data and update parameters; and(Huang [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN")
using the updated parameters, control the first neural network to generate a second synthetic data.(Huang [Col. 15 l. 12-15] "cause the generator trained using the first synthetic data and the second synthetic data to generate third synthetic data").
Regarding claim 4, the combination of Huang and Tucker teaches The system of claim 1, wherein the instructions further cause the processor to: based at least in part on the prediction incorrectly identifying the generated synthetic data, control the second neural network to execute a machine learning algorithm to calculate a loss for the second neural network and update parameters.(Huang [Col. 5 l. 24-36] "he generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN" "discrepancy between Dt(st,a) and |Dt(st+1,r)" interpreted as the loss).
Regarding claim 7, the combination of Huang and Tucker teaches The system of claim 1, wherein the instructions further cause the first neural network to generate the synthetic data by: distorting feature values of the original data to introduce noise.(Huang [Col. 6 l. 47-60] "“Z” represents random variables that are input into generator. In one or more embodiments, Z is a multi-dimensional white noise term").
Regarding claim 8, the combination of Huang and Tucker teaches The system of claim 1, wherein the instructions further cause the second neural network to: generate the prediction by alternating affine and non-linear activation functions.(Tucker [p. 5] "Our pixel-class CNN outputs class logits zijk for each pixel (i,j), rather than making a direct prediction of the pixel value. The class label cij is sampled from softmax(zij)" CNN layer is affine and softmax is nonlinear).
Regarding claim 10, Huang teaches A computer-implemented method for deep reinforcement learning, the method comprising: generating, by a first neural network implemented on a processor, synthetic data based on original data;([Col. 9 l. 57-61] "Processing circuitry 18 is configured to cause generator 30 trained with training data to generate first synthetic data (Block S122). In one or more embodiments, generator 30 trained with minibatches of data D1 (s, a, s', r) generates a batch of data D2(s, a, s', r).")
providing the original data and the generated synthetic data to a second neural network implemented on the processor;([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 9 l. 20-50] "Merge Sl (s,a) and S2 (s',r) to form a batch of D3 (s,a,s'r) Feed D3 (s,a,s'r) as real data into GAN for a training session [...] train generator 30 and discriminator 32 using the training data (Block S120). In one or more embodiments, generator 30 and discriminator 32 are trained with minibatches or portions of training data, e.g., D1 (s, a, s', r)" Huang explicitly trains the discriminator (second network) with original data D1, and feeds synthetic data S2 along with input data S1)
wherein the first neural network and the second neural network are structured to continuously learn using deep reinforcement learning including a state, an action, a reward, and a next state,([Col. 9 l. 20-45] "Repeat […] Until GAN converges [...] the pre-training procedure can be updated [...] the training data includes state (s), action (a), transitioned to state (s′) for choosing action (a), and reward (r) for choosing action (a) such that training data is written as D(s, a, s′, r), e.g., D1(s, a, s′, r)" Huang explicitly discloses all four reinforcement learning fields and Algorithm 2 repeatedly trains the GAN until convergence or timeout with continuous update when later samples become available)
wherein the state is a value of an input feature in the original data, ([Col. 9 l. 40] "the training data includes state (s)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" In Huang the state s is literally an input field/value in the original data)
the action [is continuous and] includes the synthetic data, ([Col. 9 l. 40] "the training data includes […] action (a)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" Huang discloses that synthetic data D2/D3/D4 include action values a as part of the generated tuple)
the reward is a measure of how unsuccessful the second neural network is in discriminating the original data from the synthetic data, ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly uses the reward value as a measure to determine the loss, the loss is in turned used to determine convergence, the convergence explicitly being when the discriminator reaches a point of being unsuccessful in discriminating the original data from the synthetic data)
and the next state is a next group of examples to generate a next iteration of the synthetic data based on the original data,([Col. 9 l. 35-45] "Generate a batch of D2 (s,a,s′,r) via the generator G […] Get as output a batch of S2 (s′,r) from DNN Merge S1 (s,a) and S2 (s′,r) to form a batch of D3 (s,a,s′r) […] transitioned to state (s')" Huang describes the next state s' as part of the RL tuple and discloses successive generated batches D2, D3, D4 relying on s' across iterations)
generating, by the second neural network, a prediction identifying the original data and the generated synthetic data; ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator")
and based at least in part on the prediction incorrectly identifying the generated synthetic data, exporting the generated synthetic data ([Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly trains the GAN until it converges by successfully tricking the discriminator into incorrectly identifying enough of the synthetic data, at which point the synthetic data is output/exported for experience replay data).
However, Huang does not explicitly teach the action is continuous and includes the synthetic data.
Tucker, in the same field of endeavor, teaches the action is continuous and includes the synthetic data, ([p. 3 §3] "We build on adversarial IRL (AIRL), an algorithm achieving state-of-the-art performance on simulated robotics tasks (Fu et al., 2017). The reference implementation of AIRL assumes a continuous action space and uses a fully-connected policy and reward network [...] Adversarial IRL formulates the inverse reinforcement learning problem as a GAN (Goodfellow et al., 2014). We learn a reward function fθ(s,a) for taking action a in state s and a stochastic policy π(a | s). The policy is the generator, and is trained using forward RL on the reward function fθ(s,a). The discriminator is restricted to have the special form").
Huang as well as Tucker are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, Huang as well as Tucker are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Huang with the teachings of Tucker by using a continuous action space. One of ordinary skill in the art would realize that the action space must be either discrete or continuous and Tucker explicitly acknowledges continuous action space in reinforcement learning generative adversarial networks and provides as additional motivation for combination ([p. 1 §1] “Recent deep IRL algorithms have achieved good performance on a variety of continuous control tasks”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 11, the combination of Huang and Tucker teaches The computer-implemented method of claim 10, further comprising: based at least in part on the prediction correctly identifying the generated synthetic data, executing, by the first neural network, a machine learning model to calculate a loss for the generated synthetic data (Huang [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN")
and update parameters for the generated synthetic data; and generating, by the first neural network, a second synthetic data.(Huang [Col. 15 l. 12-15] "cause the generator trained using the first synthetic data and the second synthetic data to generate third synthetic data").
Regarding claim 13, the combination of Huang and Tucker teaches The computer-implemented method of claim 10, further comprising: based at least in part on the prediction incorrectly identifying the generated synthetic data, executing, by the second neural network, a machine learning algorithm to calculate a loss for the second neural network and update parameters for the second neural network.(Huang [Col. 5 l. 24-36] "he generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN" "discrepancy between Dt(st,a) and |Dt(st+1,r)" interpreted as the loss).
Regarding claim 16, the combination of Huang and Tucker teaches The computer-implemented method of claim 10, wherein generating the prediction further comprises: alternating affine and non-linear activation functions.(Tucker [p. 5] "Our pixel-class CNN outputs class logits zijk for each pixel (i,j), rather than making a direct prediction of the pixel value. The class label cij is sampled from softmax(zij)" CNN layer is affine and softmax is nonlinear).
Regarding claim 17, the combination of Huang and Tucker teaches The computer-implemented method of claim 10, wherein generating the synthetic data further comprises: distorting feature values of the original data to introduce noise.(Huang [Col. 6 l. 47-60] "“Z” represents random variables that are input into generator. In one or more embodiments, Z is a multi-dimensional white noise term").
Regarding claim 18, the combination of Huang and Tucker teaches The computer-implemented method of claim 10, wherein the first neural network. the second neural network, and the original data are physically co-located or located within a same geographic region.(Huang [Col. 8 l. 16-18] "System 10 includes GAN 28 that includes generator 30 and discriminator 32." Huang explicitly places the generator and discriminator together inside the same system 10).
Regarding claim 19, Huang teaches One or more computer-storage memory devices embodied with executable operations that, when executed by a processor, cause the processor to:([Abstract] "A pre-training apparatus and method for reinforcement learning based on a Generative Adversarial Network (GAN) is provided" [Col. 6 l. 17-33] "Apparatus 12 includes processing circuitry 18. Processing circuitry 18 includes processor 20 and memory 22. In addition to a traditional processor and memory, processing circuitry 18 may comprise integrated circuitry for processing and/or control, e.g., one or more processors and/or processor cores ")
receive an original data group; generate, by a first neural network, a first synthetic data group based on the original data group;([Col. 9 l. 57-61] "Processing circuitry 18 is configured to cause generator 30 trained with training data to generate first synthetic data (Block S122). In one or more embodiments, generator 30 trained with minibatches of data D1 (s, a, s', r) generates a batch of data D2(s, a, s', r).")
provide the original data group and the generated first synthetic data group to a second neural network different from the first neural network; ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 9 l. 20-50] "Merge Sl (s,a) and S2 (s',r) to form a batch of D3 (s,a,s'r) Feed D3 (s,a,s'r) as real data into GAN for a training session [...] train generator 30 and discriminator 32 using the training data (Block S120). In one or more embodiments, generator 30 and discriminator 32 are trained with minibatches or portions of training data, e.g., D1 (s, a, s', r)" Huang explicitly trains the discriminator (second network) with original data D1, and feeds synthetic data S2 along with input data S1)
generate, by a second neural network, a first prediction identifying the original data group and the generated first synthetic data group;([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator")
based at least in part on the first prediction correctly identifying the generated synthetic data group: execute, by the first neural network, a first machine learning (ML) model to calculate a loss for the generated first synthetic data group, update parameters, ([Col. 5 l. 24-36] "he generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN" "discrepancy between Dt(st,a) and |Dt(st+1,r)" interpreted as the loss)
and, using the updated parameters, generate a second synthetic data group, wherein the generated second synthetic data group is a second iteration of the generated first synthetic data group based on the original data group, and([Col. 9 l. 35-45] "Generate a batch of D2 (s,a,s′,r) via the generator G […] Get as output a batch of S2 (s′,r) from DNN Merge S1 (s,a) and S2 (s′,r) to form a batch of D3 (s,a,s′r) […] transitioned to state (s')" [Col. 15 l. 14-15] "to cause the generator trained using the first synthetic data and the second synthetic data to generate third synthetic data." Huang describes the next state s' as part of the RL tuple and discloses successive generated batches D2, D3, D4 relying on s' across iterations)
execute, by the second neural network, a second ML model to calculate a loss for the second neural network and update parameters;([Col. 5 l. 24-36] "he generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN" "discrepancy between Dt(st,a) and |Dt(st+1,r)" interpreted as the loss)
provide the original data group and the generated second synthetic data group to the second neural network;([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 9 l. 20-50] "Merge Sl (s,a) and S2 (s',r) to form a batch of D3 (s,a,s'r) Feed D3 (s,a,s'r) as real data into GAN for a training session [...] train generator 30 and discriminator 32 using the training data (Block S120). In one or more embodiments, generator 30 and discriminator 32 are trained with minibatches or portions of training data, e.g., D1 (s, a, s', r)" Huang explicitly trains the discriminator (second network) with original data D1, and feeds synthetic data S2 along with input data S1)
wherein the first neural network and the second neural network are structured to continuously learn using deep reinforcement learning including a state, an action, a reward, and a next state,([Col. 9 l. 20-45] "Repeat […] Until GAN converges [...] the pre-training procedure can be updated [...] the training data includes state (s), action (a), transitioned to state (s′) for choosing action (a), and reward (r) for choosing action (a) such that training data is written as D(s, a, s′, r), e.g., D1(s, a, s′, r)" Huang explicitly discloses all four reinforcement learning fields and Algorithm 2 repeatedly trains the GAN until convergence or timeout with continuous update when later samples become available)
wherein the state is a value of an input feature in the original data group, ([Col. 9 l. 40] "the training data includes state (s)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" In Huang the state s is literally an input field/value in the original data)
the action [is continuous and] includes the first synthetic data group,([Col. 9 l. 40] "the training data includes […] action (a)" [Col. 9 l. 10-21] "/*Data Input*/ Take a batch of quadruplets Dl(s,a,s',r) from the real experience […] Put as input a slide Sl (s,a) of the batch D2 into DNN" Huang discloses that synthetic data D2/D3/D4 include action values a as part of the generated tuple)
the reward is a measure of how unsuccessful the second neural network is in discriminating the original data group from the first synthetic data group, ([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly uses the reward value as a measure to determine the loss, the loss is in turned used to determine convergence, the convergence explicitly being when the discriminator reaches a point of being unsuccessful in discriminating the original data from the synthetic data)
and the next state is a next group of examples to generate the second synthetic data group based on the second iteration of the generated first synthetic data group based on the original data group;([Col. 9 l. 35-45] "Generate a batch of D2 (s,a,s′,r) via the generator G […] Get as output a batch of S2 (s′,r) from DNN Merge S1 (s,a) and S2 (s′,r) to form a batch of D3 (s,a,s′r) […] transitioned to state (s')" Huang describes the next state s' as part of the RL tuple and discloses successive generated batches D2, D3, D4 relying on s' across iterations)
generate, by the second neural network, a second prediction identifying the original data group and the generated first synthetic data group; and([Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator")
based at least in part on the second prediction incorrectly identifying the generated second synthetic data group, export the generated second synthetic data group.([Col. 5 l. 24-36] "the generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly trains the GAN until it converges by successfully tricking the discriminator into incorrectly identifying enough of the synthetic data, at which point the synthetic data is output/exported for experience replay data).
However, Huang does not explicitly teach the action [is continuous and] includes the first synthetic data group.
Tucker, in the same field of endeavor, teaches the action [is continuous and] includes the first synthetic data group,([p. 3 §3] "We build on adversarial IRL (AIRL), an algorithm achieving state-of-the-art performance on simulated robotics tasks (Fu et al., 2017). The reference implementation of AIRL assumes a continuous action space and uses a fully-connected policy and reward network [...] Adversarial IRL formulates the inverse reinforcement learning problem as a GAN (Goodfellow et al., 2014). We learn a reward function fθ(s,a) for taking action a in state s and a stochastic policy π(a | s). The policy is the generator, and is trained using forward RL on the reward function fθ(s,a). The discriminator is restricted to have the special form").
Huang as well as Tucker are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, Huang as well as Tucker are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Huang with the teachings of Tucker by using a continuous action space. One of ordinary skill in the art would realize that the action space must be either discrete or continuous and Tucker explicitly acknowledges continuous action space in reinforcement learning generative adversarial networks and provides as additional motivation for combination ([p. 1 §1] “Recent deep IRL algorithms have achieved good performance on a variety of continuous control tasks”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 20, the combination of Huang and Tucker teaches The one or more computer-storage memory devices of claim 19, wherein the processor further: exports the generated second synthetic data group to a third ML model. (Huang [Col. 6 l. 65-67] "deep neural network (DNN) is added to learn this relation and enforce the data generated by the GAN" DNN interpreted as third ML model that generated second synthetic data group (experience data) is exported to).
Claims 3, 5, 12, and 14 are rejected under U.S.C. §103 as being unpatentable over the combination of Huang and Tucker and in further view of Ho (“Generative Adversarial Imitation Learning”, 2016).
Regarding claim 3, the combination of Huang and Tucker teaches The system of claim 2.
However, the combination of Huang and Tucker doesn't explicitly teach, wherein the instructions further cause the first neural network to update the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient.
Ho, in the same field of endeavor, teaches the instructions further cause the first neural network to update the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient ([p. 6] "GAIL alternates between an Adam [11] gradient step on w to increase Eq. (16) with respect to D, and a TRPOstep on θ to decrease Eq. (16) with respect to π (we derive an estimator for the causal entropy gradient ∇θH(πθ) in Appendix A.2). The TRPO step serves the same purpose as it does with the apprenticeship learning algorithm of Ho et al. [10]: it prevents the policy from changing too much due to noise in the policy gradient. The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function c(s,a) = logD(s,a) will move toward expert-like regions of state-action space, as classified by the discriminator.").
The combination of Huang and Tucker as well as Ho are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, the combination of Huang and Tucker as well as Ho are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Ho by using the ADAM gradient step for the discriminator. Ho provides as additional motivation for combination ([p. 6] “unlike linear apprenticeship learning algorithms, it can imitate expert policies exactly”).
Regarding claim 5, the combination of Huang and Tucker teaches The system of claim 4.
However, the combination of Huang and Tucker doesn't explicitly teach, wherein the instructions further cause the second neural network to update the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient..
Ho, in the same field of endeavor, teaches the instructions further cause the second neural network to update the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient. ([p. 6] "GAIL alternates between an Adam [11] gradient step on w to increase Eq. (16) with respect to D, and a TRPOstep on θ to decrease Eq. (16) with respect to π (we derive an estimator for the causal entropy gradient ∇θH(πθ) in Appendix A.2). The TRPO step serves the same purpose as it does with the apprenticeship learning algorithm of Ho et al. [10]: it prevents the policy from changing too much due to noise in the policy gradient. The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function c(s,a) = logD(s,a) will move toward expert-like regions of state-action space, as classified by the discriminator.").
The combination of Huang and Tucker as well as Ho are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, the combination of Huang and Tucker as well as Ho are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Ho by using the ADAM gradient step for the discriminator. Ho provides as additional motivation for combination ([p. 6] “unlike linear apprenticeship learning algorithms, it can imitate expert policies exactly”).
Regarding claim 12, the combination of Huang and Tucker teaches The computer-implemented method of claim 11.
However, the combination of Huang and Tucker doesn't explicitly teach, further comprising: updating, by the first neural network, the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient.
Ho, in the same field of endeavor, teaches The computer-implemented method of claim 11, further comprising: updating, by the first neural network, the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient.([p. 6] "GAIL alternates between an Adam [11] gradient step on w to increase Eq. (16) with respect to D, and a TRPOstep on θ to decrease Eq. (16) with respect to π (we derive an estimator for the causal entropy gradient ∇θH(πθ) in Appendix A.2). The TRPO step serves the same purpose as it does with the apprenticeship learning algorithm of Ho et al. [10]: it prevents the policy from changing too much due to noise in the policy gradient. The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function c(s,a) = logD(s,a) will move toward expert-like regions of state-action space, as classified by the discriminator.").
The combination of Huang and Tucker as well as Ho are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, the combination of Huang and Tucker as well as Ho are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Ho by using the ADAM gradient step for the discriminator. Ho provides as additional motivation for combination ([p. 6] “unlike linear apprenticeship learning algorithms, it can imitate expert policies exactly”).
Regarding claim 14, the combination of Huang and Tucker teaches The computer-implemented method of claim 13.
However, the combination of Huang and Tucker doesn't explicitly teach, further comprising: updating, by the second neural network, the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient.
Ho, in the same field of endeavor, teaches The computer-implemented method of claim 13, further comprising: updating, by the second neural network, the parameters by subtracting a gradient of the calculated loss and minimizing values in an opposite direction of the gradient.([p. 6] "GAIL alternates between an Adam [11] gradient step on w to increase Eq. (16) with respect to D, and a TRPOstep on θ to decrease Eq. (16) with respect to π (we derive an estimator for the causal entropy gradient ∇θH(πθ) in Appendix A.2). The TRPO step serves the same purpose as it does with the apprenticeship learning algorithm of Ho et al. [10]: it prevents the policy from changing too much due to noise in the policy gradient. The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function c(s,a) = logD(s,a) will move toward expert-like regions of state-action space, as classified by the discriminator.").
The combination of Huang and Tucker as well as Ho are directed towards hybrid reinforcement learning generative adversarial neural network models. Therefore, the combination of Huang and Tucker as well as Ho are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Ho by using the ADAM gradient step for the discriminator. Ho provides as additional motivation for combination ([p. 6] “unlike linear apprenticeship learning algorithms, it can imitate expert policies exactly”).
Claims 6 and 15 are rejected under U.S.C. §103 as being unpatentable over the combination of Huang and Tucker and Zhang (“Progressive Augmentation of GANs”, 2019).
Regarding claim 6, the combination of Huang and Tucker teaches provide the labeled original data and the generated synthetic data to the second neural network, and receive the prediction generated by the second neural network. (Huang [Col. 5 l. 22-24] "a discriminative model D (discriminator) that estimates the probability that a sample came from training data rather than the generator" [Col. 9 l. 20-50] "Merge Sl (s,a) and S2 (s',r) to form a batch of D3 (s,a,s'r) Feed D3 (s,a,s'r) as real data into GAN for a training session [...] train generator 30 and discriminator 32 using the training data (Block S120). In one or more embodiments, generator 30 and discriminator 32 are trained with minibatches or portions of training data, e.g., D1 (s, a, s', r)" Huang explicitly trains the discriminator (second network) with original data D1, and feeds synthetic data S2 along with input data S1).
However, the combination of Huang and Tucker doesn't explicitly teach The system of claim 1, wherein the instructions further cause the processor to: randomly assign a label to each of the original data and the generated synthetic data,
compare the prediction generated by the second neural network to the labels randomly assigned to the original data and the generated synthetic data, and based on the comparison, output a first label and a second label when the prediction is correct and incorrect respectively.
Zhang, in the same field of endeavor, teaches The system of claim 1, wherein the instructions further cause the processor to: randomly assign a label to each of the original data and the generated synthetic data,([p. 1 §1] "The key idea is to progressively augment the input of the discriminator network or its intermediate feature layers with auxiliary random bits in order to gradually increase the discrimination task difficulty (see Fig. 1) [...] As opposed to standard augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples or their features, but rather structurally appends to them. Moreover, it can also alter the input class. For instance, in the single-level augmentation the data sample or its features x are combined with a random bit s and both are provided to the discriminator")
compare the prediction generated by the second neural network to the labels randomly assigned to the original data and the generated synthetic data, and based on the comparison, output a first label and a second label when the prediction is correct and incorrect respectively.([p. 1 §1] "The key idea is to progressively augment the input of the discriminator network or its intermediate feature layers with auxiliary random bits in order to gradually increase the discrimination task difficulty (see Fig. 1) [...] As opposed to standard augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples or their features, but rather structurally appends to them. Moreover, it can also alter the input class. For instance, in the single-level augmentation the data sample or its features x are combined with a random bit s and both are provided to the discriminator" See FIG. 1 where Discriminator explicitly outputs binary label of either True or Fake based on the randomly assigned bits/labels).
The combination of Huang and Tucker as well as Zhang are directed towards generative adversarial networks. Therefore, the combination of Huang and Tucker as well as Zhang are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Zhang by randomly applying labels to each of the original data set and by explicitly outputting binary labels based on the discriminator prediction by implementing PA-GAN as the GAN in the combination of Huang and Tucker as a substitute or in combination to the standard GAN. Zhang provides as additional motivation for combination ([Abstract] “We experimentally demonstrate the effectiveness of PA-GAN across different architectures and on multiple benchmarks for the image synthesis task, on average achieving 3 point improvement of the FID score”).
Regarding claim 15, the combination of Huang and Tucker teaches The computer-implemented method of claim 10.
However, the combination of Huang and Tucker doesn't explicitly teach randomly assigning a label to each of the original data and the generated synthetic data, and providing the labeled original data and the generated synthetic data to the second neural network.
Zhang, in the same field of endeavor, teaches randomly assigning a label to each of the original data and the generated synthetic data, and providing the labeled original data and the generated synthetic data to the second neural network. ([p. 1 §1] "The key idea is to progressively augment the input of the discriminator network or its intermediate feature layers with auxiliary random bits in order to gradually increase the discrimination task difficulty (see Fig. 1) [...] As opposed to standard augmentation techniques (e.g. rotation, cropping, resizing), the proposed progressive augmentation does not directly modify the data samples or their features, but rather structurally appends to them. Moreover, it can also alter the input class. For instance, in the single-level augmentation the data sample or its features x are combined with a random bit s and both are provided to the discriminator").
The combination of Huang and Tucker as well as Zhang are directed towards generative adversarial networks. Therefore, the combination of Huang and Tucker as well as Zhang are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Zhang by randomly applying labels to each of the original data set and by explicitly outputting binary labels based on the discriminator prediction by implementing PA-GAN as the GAN in the combination of Huang and Tucker as a substitute or in combination to the standard GAN. Zhang provides as additional motivation for combination ([Abstract] “We experimentally demonstrate the effectiveness of PA-GAN across different architectures and on multiple benchmarks for the image synthesis task, on average achieving 3 point improvement of the FID score”).
Claim 9 is rejected under U.S.C. §103 as being unpatentable over the combination of Huang and Tucker as evidenced by Mathworks (“Monitor GAN Training Progress and Identify Common Failure Modes”, 2021).
Regarding claim 9, the combination of Huang and Tucker teaches The system of claim 1, wherein the generated synthetic data is exported for training an external machine learning model based at least in part on the prediction incorrectly identifying the generated synthetic data a number of times (Huang [Col. 5 l. 24-36] "he generator can be thought of as analogous to a group of counterfeiters trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. A goal of the adversarial process in GAN is to drive both the generator and discriminator to improve their methods until the counterfeits are indistinguishable from the genuine articles, i.e., until training data or real data is indistinguishable from synthesized data or data generated by the generator. In other words, ideally, the discriminator learns to capture distinguishing features of real data, which the generator learns to imitate, and the process iterates until real data and synthesized data are indistinguishable" [Col. 7 l. 45-50] "while convergence condition not met do […] using the enhancer to calculate the discrepancy between Dt(st,a) and |Dt(st+1,r) and use this to update GAN [...] synthesized experience replay data generation;" Huang explicitly trains the GAN until it converges by successfully tricking the discriminator into incorrectly identifying enough of the synthetic data, at which point the synthetic data is output/exported for experience replay data).
While it would have been obvious in view of Huang’s stated goal that GAN convergence occurs when the discriminator is sufficiently fooled, the combination of Huang and Tucker does not explicitly teach a number of times that exceeds a threshold greater than one.
Mathworks, in the same field of endeavor teaches the generated synthetic data is exported for training an external machine learning model based at least in part on the prediction incorrectly identifying the generated synthetic data a number of times that exceeds a threshold greater than one ([p. 1] "Convergence Failure Convergence failure happens when the generator and discriminator do not reach a balance during training. Discriminator Dominates This scenario happens when the generator score reaches zero or near zero and the discriminator score reaches one or near one. This plot shows an example of the discriminator overpowering the generator. Notice that the generator score approaches zero and does not recover. In this case, the discriminator classifies most of the images correctly. In turn, the generator cannot produce any images that fool the discriminator and thus fails to learn." Mathworks Matlab GAN documentation explicitly shows that convergence fails if the discriminator does not incorrectly identify the generated synthetic data a number of times that exceeds a threshold greater than one. Note that Mathworks also acknowledges that in discriminator dominated convergence failure situations the discriminator still fails to correctly identify all of the generated synthetic data.)
The combination of Huang and Tucker as well as Mathworks are directed towards generative adversarial networks. Therefore, the combination of Huang and Tucker as well as Zhang are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Huang and Tucker with the teachings of Mathworks by training the GAN based on the discriminator being fooled more than once. Mathworks describes this as not only expected but describes any situation where this does not occur as common convergence failure.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Fu (“LEARNING ROBUST REWARDS WITH ADVERSARIAL INVERSE REINFORCEMENT LEARNING”, 2018) is directed towards adversarial imitation learning with continuous action space.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights