DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 01/23/2026 have been fully considered but they are not persuasive.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 101, the applicant argues that the amended claims directed to a technical solution. Examiner respectfully agrees and withdraws the prior rejections of claims under 35 USC § 101.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 102, the arguments are directed to newly amended limitations that were not previously examined by the examiner. Therefore, applicants arguments are rendered moot. The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 5-6, 9-10, 13-14, and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, Tingwu, et al. "Nervenet: Learning structured policy with graph neural networks." International conference on learning representations. 2018 (“Wang”) in view of Kipf, Thomas, et al. "Neural Relational Inference for Interacting Systems." arXiv preprint arXiv:1802.04687 (2018) (“Kipf”) and evidenced by Lesort, Timothée, et al. "State Representation Learning for Control: An Overview." arXiv preprint arXiv:1802.04181 (2018) (“Lesort”).
In regards to claim 1 and analogous claims 9 and 17,
Wang teaches A reinforcement learning method,
(Wang, Abstract, “We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by multi-layer perceptrons (MLPs) which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet [A reinforcement learning method] to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent’s policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent.”)
PNG
media_image1.png
370
108
media_image1.png
Greyscale
Wang teaches [obtaining, by executing a structure learning model, a structure graph] based on which an intelligent agent interacts with an environment wherein the structure graph comprises structure information of the environment or the intelligent agent,
(Wang, Fig. 2, “Figure 2: In this figure, we use Walker-Ostrich as an example of NerveNet. In the input model, for each node, NerveNet fetches the corresponding elements [based on which an intelligent agent interacts with an environment
wherein the structure graph comprises structure information of the environment or the intelligent agent; ie agent’s position and velocity in the environment] from the observation vector.”)
Wang teaches inputting a current state of the environment and the structure graph to a policy function of the intelligent agent, wherein an action is generated in response to the current state and the structure graph using the policy function,
(Wang, Section 2, “We formulate the locomotion control problems as an infinite-horizon discounted Markov decision process (MDP). To fully describe the MDP for continuous control problems which include locomotion control, we define the state space or observation space as S and action space as A. To interact with the environments, the agent generates its stochastic policy πθ(a τ |s τ ) based on the current state s τ ∈ S [inputting a current state of the environment and the structure graph to a policy function of the intelligent agent], where a τ ∈ A is the action [wherein an action is generated in response to the current state and the structure graph using the policy function] and θ are the parameters of the policy function. The environment on the other hand, produces a reward r(s τ , aτ ) for the agent, and the agent’s objective is to find a policy that maximizes the expected reward.”)
Wang teaches and the policy function of the intelligent agent is a graph neural network distinct from the structure learning model;
(Wang, Section 2.2, “We now turn to NerveNet which parametrizes the policy with a Graph Neural Network [policy function of the intelligent agent is a graph neural network distinct from the structure learning model; wherein Examiner notes that the model of Kipf is relied upon to take the observations to derive the state/input model and the policy of Wang determines how the agent is going to further move in the environment].”)
Wang teaches outputting the action to the environment by using the intelligent agent;
(Wang, Section 2.2, “We denote the set of nodes which are assigned controllers for the actuators as O. For each such node, a MLP takes its final state vectors h T u∈O as input and produces the mean of the action of the Gaussian policy for the corresponding actuator. For each output node u ∈ O, we define its output type as qu. Different sharing schemes are available for the instance of MLP Oqu , for example, we can force the nodes with similar physical structure to share the instance of MLP. For example, in Fig. 1, two LeftHip nodes have a shared controller. Therefore, we have the following output model:
PNG
media_image2.png
33
557
media_image2.png
Greyscale
where µu∈O is the mean value for action applied on each actuator [outputting the action to the environment by using the intelligent agent; wherein applying an action on an actuator is the agent performing an action in the environment]. In practice, we found that we can force controllers of different output types to share one unified controller, while not hurting the performance.”)
Wang teaches obtaining, from the environment by using the intelligent agent, a next state and reward data in response to the action; and training the intelligent agent through reinforcement learning based on the reward data.
(Wang, Section 2.2.1, “We formulate the locomotion control problems as an infinite-horizon discounted Markov decision process (MDP). To fully describe the MDP for continuous control problems which include locomotion control, we define the state space or observation space as S and action space as A. To interact with the environments, the agent generates its stochastic policy πθ(a τ |s τ ) based on the current state s τ ∈ S, where a τ ∈ A is the action and θ are the parameters of the policy function [obtaining, from the environment by using the intelligent agent, a next state and reward data in response to the action]. The environment on the other hand, produces a reward r(s τ , aτ ) for the agent, and the agent’s objective is to find a policy that maximizes the expected reward [training the intelligent agent through reinforcement learning based on the reward data].”)
However, Wang does not explicitly teach obtaining, by executing a structure learning model, a structure graph… and the structure information is obtained through learning, and wherein the structure learning model learns the structure graph through historical interaction data of the environment by performing automated inference of structure relationships among entities represented in the historical interaction data;
Kipf teaches comprising: obtaining, by executing a structure learning model, a structure graph [based on which an intelligent agent interacts with an environment],
(Kipf, Abstract, “In this work, we introduce the neural relational inference (NRI) model [by executing a structure learning model]: an unsupervised model that learns to infer interactions while simultaneously learning the dynamics purely from observational data. Our model takes the form of a variational auto-encoder, in which the latent code represents the underlying interaction graph [obtaining… a structure graph] and the reconstruction is based on graph neural networks.”
PNG
media_image3.png
285
397
media_image3.png
Greyscale
)
Kipf teaches and the structure information is obtained through learning, and wherein the structure learning model learns the structure graph through historical interaction data of the environment by performing automated inference of structure relationships among entities represented in the historical interaction data;
(Kipf, Section 3., “Our NRI model consists of two parts trained jointly [the structure information is obtained through learning]: An encoder that predicts the interactions given the trajectories [by performing automated inference; ie predicting interactions], and a decoder that learns the dynamical model given the interaction graph.
More formally, our input consists of trajectories of N objects. We denote by xt i the feature vector of object vi at time t, e.g. location and velocity. We denote by xt = {xt 1,...,xt N} the set of features of all N objects at time t, and we denote by xi = (x1 i,...,xT i ) the trajectory of object i, where T is the total number of time steps [of structure relationships among entities ie nodes of the agent of Chen represented in the historical interaction data; as the agent moves in the environment as measured by positional information, velocity, etc (observation vector)]. Lastly, we mark the whole trajectories by x = (x1,...,xT). We assume that the dynamics can be modeled by a GNN given an unknown graph z where zij represents the discrete edge type between objects vi and vj. The task is to simultaneously learn to predict the edge types and learn the dynamical model in an unsupervised way.” See Fig. 7 for a latent graph of a body (skeleton) moving observed from motion capture data
PNG
media_image4.png
294
398
media_image4.png
Greyscale
)
Wang is considered to be analogous to the claimed invention because they are in the same field of reinforcement learning using graph neural networks. Kipf is considered to be analogous to the claimed invention because they are reasonably pertinent to the problem the inventor faced of inferring interactions from mere observational data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Kipf in order to provide a neural relational inference model that can infer interactions purely from observational data as doing so could provide an accurate ground-truth interpretable structure, particularly in the realm of state representation learning as disclosed by Lesort, wherein the NRI model of Kipf interpreted as state representation learning, could improve performance and speed of learning the policy of Wang in reinforcement learning (Kipf, Abstract, “Interacting systems are prevalent in nature, from dynamical systems in physics to complex societal dynamics. The interplay of components can give rise to complex behavior, which can often be explained using a simple model of the system’s constituent parts. In this work, we introduce the neural relational inference (NRI) model: an unsupervised model that learns to infer interactions while simultaneously learning the dynamics purely from observational data. Our model takes the form of a variational auto-encoder, in which the latent code represents the underlying interaction graph and the reconstruction is based on graph neural networks. In experiments on simulated physical systems, we show that our NRI model can accurately recover ground-truth interactions in an unsupervised manner. We further demonstrate that we can find an interpretable structure and predict complex dynamics in real motion capture and sports tracking data.”) (Lesort, Abstract, “The representation is learned to capture the variation in the environment generated by the agent’s actions; this kind of representation is particularly suit able for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning.”)
In regards to claim 2,
Wang and Kipf teaches The method according to claim 1,
Kipf teaches wherein the obtaining, by executing a structure learning model, a structure graph comprises: obtaining the historical interaction data of the environment;
inputting the historical interaction data to a structure learning model; and
learning the structure graph from the historical interaction data by using the structure learning model.
(Kipf, Section 3., “Our NRI model consists of two parts trained jointly: An encoder that predicts the interactions given the trajectories [learning the structure graph from the historical interaction data by using the structure learning model], and a decoder that learns the dynamical model given the interaction graph.
More formally, our input consists of trajectories of N objects. We denote by xt i the feature vector of object vi at time t [obtaining the historical interaction data of the environment; inputting the historical interaction data to a structure learning model], e.g. location and velocity. We denote by xt = {xt 1,...,xt N} the set of features of all N objects at time t, and we denote by xi = (x1 i,...,xT i ) the trajectory of object i, where T is the total number of time steps. Lastly, we mark the whole trajectories by x = (x1,...,xT). We assume that the dynamics can be modeled by a GNN given an unknown graph z where zij represents the discrete edge type between objects vi and vj. The task is to simultaneously learn to predict the edge types and learn the dynamical model in an unsupervised way.”)
In regards to claim 5,
Wang and Kipf teach The method according to claim 2,
Kipf teach wherein the structure learning model comprises any one of: a neural interaction inference model, a Bayesian network, or a linear non-Gaussian acyclic graph model.
(Kipf, Abstract, “In this work, we introduce the neural relational inference (NRI) model: an unsupervised model that learns to infer interactions [a neural interaction inference model ]…”)
In regards to claim 6,
Wang and Kipf teach The method according to claim 1,
Wang teaches wherein the environment is a robot control scenari0.
(Wang, Section 4.2, “Snakes We also create a snake-like agent which is common in robotics, Crespi & Ijspeert (2008). We design the Snake environment [wherein the environment is a robot control scenari0] based on the Swimmer model in Gym. The goal of the agent is to move as fast as possible. For details of the environment, please see the schematic figure 16.”)
Claim 9 and 17 are rejected on the same rationale under 35 U.S.C. 103 as claim 1.
Claim 10 is rejected on the same rationale under 35 U.S.C. 103 as claim 2.
Claim 13 is rejected on the same rationale under 35 U.S.C. 103 as claim 5.
Claim 14 is rejected on the same rationale under 35 U.S.C. 103 as claim 6.
In regards to claim 18,
Wang and Kipf teach The method according to claim 1,
Kipf teaches wherein the structure learning model comprises a Bayesian network, a linear non-Gaussian acyclic graph model, and/or a neural interaction inference model.
(Kipf, Abstract, “In this work, we introduce the neural relational inference (NRI) model: an unsupervised model that learns to infer interactions [a neural interaction inference model ]…”)
Claim(s) 3-4, 7, 11-12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Kipf in further view of D. Huang and S. Lee, "A Self-Play Policy Optimization Approach to Battling Pokémon," 2019 IEEE Conference on Games (CoG), London, UK, 2019, pp. 1-4, doi: 10.1109/CIG.2019.8848014 (“Huang”)
In regards to claim 3,
Wang and Kipf teaches The method according to claim 2,
Huang teaches wherein before the inputting the historical interaction data to a structure learning model, the method further comprises: filtering the historical interaction data by using a mask, wherein impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask.
(Huang, Section III A., “Because not every action is valid in every state (for example, a switch to a Pokemon is invalid if that ´ Pokemon is already fainted), we need to make sure our ´ agent has zero probability of taking invalid actions [impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask]. To do this, we take a mask s ∈ {0, 1} n as part of the input [filtering the historical interaction data by using a mask], and renormalize probabilities to obtain π: πi = siπ 0 i sTπ0 .”; wherein a mask is used to filter out invalid actions)
Wang and Huang are both considered to be analogous to the claimed invention because they are in the same field of reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Huang in order to provide an application of reinforcement learning to a competitive gaming environment (Pokemon battles) and a mask to ensure the agent does not take any invalid actions (Huang, Section III A., “Because not every action is valid in every state (for example, a switch to a Pokemon is invalid if that ´ Pokemon is already fainted), we need to make sure our ´ agent has zero probability of taking invalid actions.”)
In regards to claim 4,
Wang in view of Kipf and Huang teach The method according to claim 2,
Kipf teaches wherein the structure learning model calculates a loss function by using a mask, impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask, and the structure learning model learns the structure graph based on the loss function.
(Kipf, Section 3., “Our NRI model consists of two parts trained jointly [the structure learning model learns the structure graph based on the loss function]: An encoder that predicts the interactions given the trajectories, and a decoder that learns the dynamical model given the interaction graph…
Section 3.4, “If we look at the ELBO, Eq.3, there construction loss term has the form T t=1log[p(xt|xt−1,z)] [wherein the structure learning model calculates a loss function ]…”)
However, Kipf does not explicitly teach by using a mask, impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask
Huang teaches using a mask, impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask
PNG
media_image5.png
158
76
media_image5.png
Greyscale
(Huang, Section III A., “Because not every action is valid in every state (for example, a switch to a Pokemon is invalid if that ´ Pokemon is already fainted), we need to make sure our ´ agent has zero probability of taking invalid actions [impact of an action of the intelligent agent on the historical interaction data is eliminated using the mask]. To do this, we take a mask s ∈ {0, 1} n as part of the input, and renormalize probabilities to obtain π: πi = siπ 0 i sTπ0 .”; wherein a mask is used to filter out invalid actions))
In regards to claim 7,
Wang and Kipf teach The method according to claim 1,
Huang teaches wherein the environment is a gaming environment comprising structure information.
(Huang, Section III A., “The input to the neural network is the current state of the game, from the point of view of the player. This is a complex multi-level tree-like structure: 1) The battle consists of two teams, along with weather effects. 2) Each team consists of six Pokemon ´ , along with side conditions (e.g. entry hazards, Reflect). 3) Each Pokemon ´ has many features. Table I contains a partial list.”)
Claim 11 is rejected on the same rationale under 35 U.S.C. 103 as claim 3.
Claim 12 is rejected on the same rationale under 35 U.S.C. 103 as claim 4.
Claim 15 is rejected on the same rationale under 35 U.S.C. 103 as claim 7.
Claim(s) 8 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Kipf in further view of Z. Xu, Y. Wang, J. Tang, J. Wang and M. C. Gursoy, "A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs," 2017 IEEE International Conference on Communications (ICC), Paris, France, 2017, pp. 1-6, doi: 10.1109/ICC.2017.7997286 (“Xu”)
In regards to claim 8,
Wang and Kipf teach The method according to claim 1,
Xu teaches wherein the environment is a scenario of optimizing an engineering parameter of a multi-cell base station.
PNG
media_image6.png
466
446
media_image6.png
Greyscale
(Xu, Section III., “We consider a single cell in a typical cloud RAN, which consists of a set of R RRHs R = {1, 2, ··· , R} and a set of U users U = {1, 2, ··· , U}. As shown in Fig. 1, in each decision epoch tk, the network reports the demands of associated users and the current states (active, sleep, etc) of all RRHs to the DRL agent. Moreover, according to the result of the last executed action, the reward can be calculated by the DRL agent (which is explained later). Here, we assume that all RRHs and users are equipped with a single antenna. The proposed solution can be easily generalized to the multiantenna case as shown in [6]. All RRHs are connected to a cloud BBU pool, thus the information of RRHs can be shared in a centralized manner, and all users can access all RRHs in the cell. The DRL agent will make an action decision ak based on the system state sk, turn on or into sleep certain RRH(s), and allocate a beamforming weight wr,u from every RRH r to each user u.”)
Wang and Xu are both considered to be analogous to the claimed invention because they are in the same field of reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Xu in order to provide an application of reinforcement learning to cellular networks to enable power-efficient resource allocation (Xu, Abstract, “Cloud Radio Access Networks (RANs) have become a key enabling technique for the next generation (5G) wireless communications, which can meet requirements of massively growing wireless data traffic. However, resource allocation in cloud RANs still needs to be further improved in order to reach the objective of minimizing power consumption and meeting demands of wireless users over a long operational period. Inspired by the success of Deep Reinforcement Learning (DRL) on solving complicated control problems, we present a novel DRL-based framework for power-efficient resource allocation in cloud RANs. Specifically, we define the state space, action space and reward function for the DRL agent, apply a Deep Neural Network (DNN) to approximate the action-value function, and formally formulate the resource allocation problem (in each decision epoch) as a convex optimization problem. We evaluate the performance of the proposed framework by comparing it with two widely-used baselines via simulation. The simulation results show it can achieve significant power savings while meeting user demands, and it can well handle highly dynamic cases.”)
Claim 16 is rejected on the same rationale under 35 U.S.C. 103 as claim 8.
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Kipf in further view of Lesort.
In regards to claim 19,
Wang and Kipf teach The method according to claim 1,
Lesort teaches wherein use of the structure graph learned by the structure learning model reduces a number of reinforcement learning iterations required for convergence of the policy function.
(Lesort, Abstract, “Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning [wherein use of the structure graph learned by the structure learning model] (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent’s actions; this kind of representation is particularly suit able for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed [reduces a number of reinforcement learning iterations required for convergence of the policy function] in policy learning algorithms such as reinforcement learning.”)
(Lesort, Section 1., “The objective of SRL is to take advantage of time steps, actions, and optionally rewards, to transform observations into states: a vector of a reduced set of the most representative features that is sufficient for efficient policy learning.”)
Lesort is considered to be analogous to the claimed invention because they are in the same field of state representation learning to improve reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang and Kipf to incorporate the teachings of Lesort in order to justify the use of the NRI model of Kipf as a state representation learning method to improve performance and speed in the policy learning of Wang (Lesort, Abstract, “The representation is learned to capture the variation in the environment generated by the agent’s actions; this kind of representation is particularly suit able for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US Pub No. US20200272905A1 GE teaches Artificial neural network compression via iterative hybrid reinforcement learning approach
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights