DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/27/2026 has been entered.
Response to Arguments
Applicant's arguments filed 01/06/2026 have been fully considered and they are partially persuasive.
Regarding applicant’s remarks directed to the rejection of claims under 35 USC § 103, the arguments are directed to newly amended limitations that were not previously examined by the examiner. Therefore, applicants arguments are rendered moot. The examiner refers to the rejection under 35 USC § 103 in the current office action for more details.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 5-6, 10, 13, 18-19 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Kong, Weiren, et al. "Maneuver strategy generation of UCAV for within visual range air combat based on multi-agent reinforcement learning and target position prediction." Applied Sciences 10.15 (2020): 5198. (“Kong”) in view of Hu, Zijian, et al. "Deep reinforcement learning approach with multiple experience pools for UAV’s autonomous motion planning in complex unknown environments." Sensors 20.7 (2020): 1890. (“Hu”) in further view of Vinitsky, Eugene, et al. "Robust reinforcement learning using adversarial populations." arXiv preprint arXiv:2008.01825 (2020). (“Vinitsky”)
In regards to claim 1,
Kong teaches A method, comprising: training a machine learning algorithm to control a first aircraft in an environment that comprises the first aircraft and a second aircraft by: determining a first-aircraft action for the first aircraft to take within the environment using the machine learning algorithm;
(Kong, Section 4.1, “The actor’s input is the state vector 𝒮 described in Section 3.3, and the output is the action vector 𝒂 described in Section 3.2 [determining a first-aircraft action for the first aircraft to take within the environment using the machine learning algorithm]. Two neural networks are used in the actor, one as the target network and the other as the online network, in order to improve the stability of the learning process. The structures of the two networks are the same, but the update methods are different. The online network uses the deterministic policy gradient for updating, and the target network copies the parameters of the online network through soft updates.”)
Kong teaches sending the first-aircraft action to a simulated environment;
(Kong, Figure 1 teaches sending the first-aircraft action to a simulated environment ie Air Combat Simulation
PNG
media_image1.png
320
605
media_image1.png
Greyscale
, Section 5, “The platform was able to simulate the 1-vs.-1 WVR air combat in certain airspaces, use the dynamic equation with three degrees of freedom to simulate the flight characteristics of UCAV, and set the performance parameters of both the UCAVs. The platform was also able to reserve the air combat strategy interface of the two UCAVs, thereby obtaining the UCAV control signals from the external environment. The platform could also support human control, where the interface received input from a keyboard or hands-on throttle and stick system (HOTS).
Figure 4 depicts the main interface of the air combat simulation platform [a simulated environment]. The interface presents a real-time combat situation of the UCAVs, and the five positions on the screen display the health value, altitude and speed, radar image, attitude angle, and all of the speed components.
PNG
media_image2.png
411
672
media_image2.png
Greyscale
”)
Kong teaches generating randomly-sampled values for a set of parameters of the second aircraft, wherein the randomly-sampled values comprise values different from predetermined fixed values for the set of parameters of the second aircraft,
(Kong, Section 5.1.4, “For experimental comparison, two different UCAV performance parameters were considered in this study. The variation of the performance capabilities focus on the following parameters: maximum thrust and maximum roll rate, as presented in Table 1. The dominant UCAV performance parameters are called the “advantage” parameters, and the non-dominant UCAV performance parameters are called the “disadvantage” parameters. The mass of both the UCAVs is same and equal to 10,000 kg.
PNG
media_image3.png
112
510
media_image3.png
Greyscale
…
Section 5.2.1 “When training the air combat countermeasures, the initial states of the two UCAVs are randomly generated according to the four situations described in Section 5.1.3. At the same time, to prevent the two UCAVs from falling into the “boring” Nash equilibrium of “biting the tail” during training, the performance parameters of the two UCAVs to be different from one another: one UCAV uses the “advantage” performance parameters, and the other UCAV uses the “disadvantage” performance parameters [generating randomly-sampled values for a set of parameters of the second aircraft ie initializing the respective aircrafts with either the “disadvantage” or “advantage” performance parameters, wherein the randomly-sampled values comprise values different from predetermined fixed values for the set of parameters of the second aircraft; wherein the performance parameters are not fixed to the aircrafts but randomly determined to generate different situations].”)
Kong teaches sending the randomly-sampled values to the simulated environment and to the machine learning algorithm;
(Kong, “Section 5.2.1 “When training the air combat countermeasures, the initial states of the two UCAVs are randomly generated according to the four situations described in Section 5.1.3. At the same time, to prevent the two UCAVs from falling into the “boring” Nash equilibrium of “biting the tail” during training, the performance parameters of the two UCAVs to be different from one another: [sending the randomly-sampled values to the simulated environment..; wherein initializing the states of the UCAV is interpreted to be sending the randomly sampled values to the simulated environment]”)
(Kong, Section 4.1, “...The actor’s input is the state vector 𝒮 described in Section 3.3…”)
(Kong, Section 3.3, “Notably, 𝑣1 and 𝑣2 are the current speeds of both UCAVs, 𝜙1 and 𝜙2 are the current heading angles of both UCAVs, and 𝜓1 and 𝜓2 are the current bank angles of both UCAVs. The bank angle is limited based on the maximum capabilities of the actual UCAV [sending the randomly-sampled values… to the machine learning algorithm; wherein section 3.3 describes the state vector/state space of the UAV (which is provided as input to the respective actor) and Hu is relied upon to teach the maximum capabilities of the respective UAV to have advantage/disadvantage performance parameters of maximum flight speed], and based on the need to limit the maximum turning capabilities of the UCAV, the heading angle is allowed to take any value between ±180∘.”)
Kong teaches after sending the first-aircraft action and the randomly-sampled values to the simulated environment, receiving an observation of the simulated environment and a reward signal at the machine learning algorithm,
(Kong, Algorithm 2 teaches executing the actions from the first and second aircrafts and providing an observation and reward in line 9 wherein the first and second aircraft were initialized in line 5 as aforementioned, “
PNG
media_image4.png
299
689
media_image4.png
Greyscale
”)
Kong teaches wherein the observation of the simulated environment comprises information related to the simulated environment after the first aircraft has taken the first-aircraft action and the second aircraft has taken a second-aircraft action based on the randomly-sampled values,
(Kong, Algorithm 2 line 9 teaches new state information S’base as information related to the simulated environment after the executed actions)
Kong teaches and wherein the reward signal indicates a score of performance of the first-aircraft action by the first aircraft based on one or more dynamic and kinematic properties of the second aircraft within the simulated environment;
(Kong, Section 3.1, “In Figure 2, 𝜆1 is the antenna train angle (ATA) of UCAV1, which is the angle between the LOS vector and UCAV1’s velocity vector. 𝜖1 is the aspect angle (AA) of UCAV1, which is the angle between the LOS vector, and UCAV2’s velocity vector. 𝜖1 can be obtained in terms of the velocity of vector 𝑽𝟏 and the line-of-sight (LOS) vector 𝝆, where 𝝆 denotes the LOS vector between UCAV1 and UCAV2. The ATA and AA are allowed to take any value between ±180∘. ATA and AA can be obtained from Equations (6) and (7).
PNG
media_image5.png
86
419
media_image5.png
Greyscale
…This article considers a 1-vs.-1 WVR air combat as a ZSMG problem. Therefore, the rewards of both the UCAVs should be opposite. According to the objective of the 1-vs.-1 WVR air combat, the reward function can be designed using Equation (8), as follows: [wherein the reward signal indicates a score of performance of the first-aircraft action by the first aircraft based on one or more dynamic and kinematic properties wherein speeds are dynamic and kinematic properties of the second aircraft within the simulated environment]
PNG
media_image6.png
79
533
media_image6.png
Greyscale
”)
Kong teaches and updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the randomly-sampled values.
(Kong, Algorithm 2 teaches updating the actor (line 20) based on the observation and reward signal (line 9), and the randomly-sampled values (line 5)
PNG
media_image7.png
488
681
media_image7.png
Greyscale
)
However, Kong does not explicitly teach wherein the randomly-sampled values comprise a respective randomly- sampled value for each parameter of the set of parameters, and wherein the set of parameters comprise one or more of a maximum weapon range, a minimum turn rate, a maximum aircraft speed, or a maximum sensor range
Hu teaches and wherein the set of parameters comprise one or more of a maximum weapon range, a minimum turn rate, a maximum aircraft speed, or a maximum sensor range;
(Hu, Section 5, “To achieve conditions similar to real-world conditions, this study designed a 3D simulation environment based on the parameters of the real UAV. At present, countries around the world have made progress in the research of UAVs and created advanced UAVs. These advanced UAVs can be used in many different tasks such as: disaster monitoring, anti-smuggling, environmental protection, meteorological observation, forest fire prevention and geological survey, etc. The ranges of some main parameter of these advanced UAVs are shown in the Table 1: [wherein the set of parameters comprise one of a maximum aircraft speed; wherein an exemplary advantage performance parameter could be 370 km/h and an exemplary disadvantage performance parameter could be 220 km/h]
PNG
media_image8.png
158
682
media_image8.png
Greyscale
”)
Vinitsky teaches wherein the randomly-sampled values comprise a respective randomly- sampled value for each parameter of the set of parameters,
(Vinitsky, Abstract, “We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training [wherein the randomly-sampled values comprise a respective randomly- sampled value for each parameter of the set of parameters; wherein the context of Kong and Hu and Vinitsky, it would be a population of second aircrafts with different performance parameters, such as max speed, would be sampled from a uniform distribution (thus, randomly sampled); Examiner further notes that a set of parameter can merely include the respective max speed].”)
Kong and Hu are both considered to be analogous to the claimed invention because they are in the same field of aerial simulation and reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong to incorporate the teachings of Hu in order to provide an environment with parameters representing a real UAV to achieve conditions similar to real-world conditions, particularly with different levels of UAVs from countries around the world. (Hu, Section 5, “To achieve conditions similar to real-world conditions, this study designed a 3D simulation environment based on the parameters of the real UAV. At present, countries around the world have made progress in the research of UAVs and created advanced UAVs.”)
Kong and Vinitsky are both considered to be analogous to the claimed invention because they are in the same field of multi-agent reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong and Hu to incorporate the teachings of Vinitsky in order to provide a population-based augmentation in order to yield robustness to dynamic variations under standard parameterizations of the adversary (Vinitsky, Abstract, Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by newadversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across robotics benchmarks that the use of an adversarial population results in a more robust policy that also improves out-of-distribution generalization. Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.”)
Claim 5 is rejected on the same rationale under 35 U.S.C. 103 as claim 1 wherein the simulator is the simulated environment.
In regards to claim 6,
Kong and Hu and Vinitsky teach The method of claim 5,
Kong teaches wherein the simulator is configured to: Receive actions from both the first and second aircraft;
(Kong, Figure 1 teaches Receive actions from both the first and second aircraft ie Air Combat Simulation receiving actions a1, a2
PNG
media_image1.png
320
605
media_image1.png
Greyscale
”)
Kong teaches Receive the randomly-sampled values;
(Kong, “Section 5.2.1 “When training the air combat countermeasures, the initial states of the two UCAVs are randomly generated according to the four situations described in Section 5.1.3. At the same time, to prevent the two UCAVs from falling into the “boring” Nash equilibrium of “biting the tail” during training, the performance parameters of the two UCAVs to be different from one another: [Receive the randomly-sampled values; wherein initializing the states of the UCAV is interpreted to be sending the randomly sampled values to the simulated environment]”)
Kong teaches determine a state of the simulated environment based on the received actions and the randomly-sampled values; and determine the information related to the simulated environment after the first and second aircraft have taken subsequent actions based on the state of the simulated environment.
(Kong, Algorithm 2 line 9 teaches new state information S’base as information related to the simulated environment after the executed actions
PNG
media_image4.png
299
689
media_image4.png
Greyscale
)
In regards to claim 10,
Kong and Hu and Vinitsky teaches The method of claim 1,
Kong teaches wherein training the machine learning algorithm to control the first aircraft comprises: training the machine learning algorithm to control the first aircraft using a plurality of scenarios related to interactions between the first and second aircraft within the simulated environment,
(Kong Section 4.1, “The critic’s inputs are the state vector 𝒮 and two action vectors 𝑎1,𝑎2, and the output is the 𝑄(𝑠,𝑎1,𝑎2) value. Similar to the actor, there are two neural networks in the critic, one for the target network and the other for the online network. The update method is also similar to that used in the case of the actor. However, the difference lies in that the input during updating is sampled from the prioritized replay memory [training the machine learning algorithm to control the first aircraft using a plurality of scenarios related to interactions between the first and second aircraft within the simulated environment].”; wherein the AI agent trains using replay memory (critic updates actor; see figure 3), which is the plurality of scenarios with associated interactions between the first aircraft and second aircraft that is used to learn from to further control the first aircraft in subsequent scenarios)
PNG
media_image9.png
378
648
media_image9.png
Greyscale
Kong teaches wherein the plurality of scenarios are arranged so that a first scenario precedes a second scenario in the plurality of scenarios,
(Kong, Algorithm 2 lines 15-16 teaches storing a scenario in the replay buffer
PNG
media_image10.png
36
466
media_image10.png
Greyscale
Section 4.4
“Prioritized experience replay memory (PERM) is a method of sampling based on experience replay and was proposed by Google DeepMind (London, UK) [26]. Experience replay enables online reinforcement learning agents to remember and reuse experiences from the past. In previous studies, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays the transitions in the same frequency that they were originally experienced in, regardless of their significance. PERM can sample experiences, more frequently prioritize to replay critical transitions [wherein the plurality of scenarios are arranged ie prioritized so that a first scenario precedes a second scenario in the plurality of scenarios], and therefore learn more efficiently.”)
Kong teaches the first scenario involving a first range of options to control the first aircraft, the second scenario involving a second range of options to control the first aircraft, and wherein the second range of options includes more options than the first range of options.
(Kong, Section 3.1, “The two players in a 1-vs.-1 WVR aerial combat engagement, referred to as UCAV1 and UCAV2, are assumed to be on the same horizontal plane. The objective of each player is to reach the tail of the adversary and track it in a stable manner, in order to satisfy the shooting condition of the guns [wherein the second range of options includes more options than the first range of options; wherein scenarios where the first aircraft meets the shooting condition (thus, this is an additional option provided in these scenarios) and destroys the second aircraft would be interpreted to be prioritized in the replay buffer as the goal of the agent is to take down the opponent; similarly, scenarios where the shooting condition is not met by the first aircraft would not be prioritized as it means the first aircraft lost].”)
In regards to claim 13,
Kong and Hu and Vinitsky teaches The method of claim 1,
Kong teaches wherein training the machine learning algorithm further comprises: training the machine learning algorithm for a first training session;
(Kong, Algorithm 2 line 4 teaches iteration through episodes wherein an episode is a training session)
Kong teaches after training the machine learning algorithm for the first training session, saving the machine learning algorithm as a previous version of the machine learning algorithm;
(Kong, Algorithm 2 lines 15-16 teaches storing a scenario in the replay buffer
PNG
media_image10.png
36
466
media_image10.png
Greyscale
”)
Kong teaches and after saving the previous version of the machine learning algorithm, continuing training of the machine learning algorithm for a second training session,
(Kong, Algorithm 2 line 4 teaches iteration through episodes wherein an episode is a training session)
Kong teaches wherein the machine learning algorithm determines actions for the first aircraft to take within the environment during the second training session,
(Kong, Section 4.1, “The actor’s input is the state vector 𝒮 described in Section 3.3, and the output is the action vector 𝒂 described in Section 3.2.”)
Kong teaches and wherein the previous version of the machine learning algorithm determines actions for the second aircraft to take within the environment during the second training session.
(Kong, Section 2.3, “Concretely speaking, the study considers the two-player ZSMG as an example of a multi-agent environment and assumes 𝜇𝑎1 to be the agent policy and 𝜇𝑎2 to be the opponent policy. Subsequently, we can write the gradient of the expected return for the agent and opponent with polices 𝜇𝑎1 and 𝜇𝑎2, 𝐽𝑎1(𝜃𝑎1)=𝔼𝑠∼𝝁𝒂𝟏,𝝁𝒂𝟐[𝐺𝑎1], 𝐽𝑎2(𝜃𝑎2)=𝔼𝑠∼𝝁𝒂𝟏,𝝁𝒂𝟐[𝐺𝑎2] as:
∇𝜃𝑎1𝐽𝑎1(𝜃𝑎1)=𝔼𝑠∼𝒟[∇𝜃𝝁𝑎1(𝑠)∇𝑎1𝑄𝝁𝒂𝟏(𝑠,𝑎1,𝑎2)|𝑎=𝝁𝒂𝟏(𝑠)]
(5)
where, 𝑄𝝁𝒂𝟏(𝑠,𝑎1,𝑎2) is a centralized action-value function of an agent that uses the agent and opponent as input and 𝑄𝝁𝒂𝟐(𝑠,𝑎1,𝑎2) is a centralized action-value function of the opponent. The experience replay buffer 𝒟 contains the tuples (𝑠,𝑠′,𝑎1,𝑜2,ℛ) recording experiences of the agent and opponent, where 𝑠′ denotes the next state from s after taking actions (𝑎1,𝑎2). Centralized action–value functions are updated using the temporal-difference learning [wherein the previous version of the machine learning algorithm determines; see eq. (5)]. Taking UCAV 1-vs.-1 WVR air combat as an application scenario, both UCAVs will have a Q network and a policy network, respectively. The input of the Q network is the flight status, air combat situation features and action vector of both UCAVs, and the output is Q value of own UCAV. The input of the policy network is the flight status, and the air combat situation features of both UCAVs, and the output is the action vector of own UCAV [actions for the second aircraft to take within the environment during the second training session].”)
Claim 18 is rejected on the same rationale under 35 U.S.C. 103 as claim 5.
Claim 19 is rejected on the same rationale under 35 U.S.C. 103 as claim 6.
Claim 22 is rejected on the same rationale under 35 U.S.C. 103 as claim 5.
Claim(s) 2-3, 15-16 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Kong in view of Hu and Vinitsky in further view of S. You, M. Diao and L. Gao, "Deep Reinforcement Learning for Target Searching in Cognitive Electronic Warfare," in IEEE Access, vol. 7, pp. 37432-37447, 2019 (“You”)
In regards to claim 2,
Kong and Hu and Vinitsky teach The method of claim 1,
You teaches wherein training the machine learning algorithm further comprises: generating a normalization of the randomly-sampled values,
(You, Section V a.2., “State normalization is an effective means to accelerate the convergence of the DRL algorithm. In the environment of a sparse state space with different scales of state values, the importance of state normalization is self-evident. In this work, based on Equation 15, we present the following state normalization approach:
PNG
media_image11.png
24
155
media_image11.png
Greyscale
wherein Vmax can be the maximum flight speed as provided by Kong in view of Hu)
Thus, Kong in view of Hu with the normalization of You teaches and wherein updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the randomly-sampled values comprises updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the normalization.
(Kong, Algorithm 2 teaches updating the actor (line 20) based on the observation and reward signal (line 9), and the randomly-sampled values wherein the normalization would be reflected in the state vector/space ie lines 5 and 13
PNG
media_image7.png
488
681
media_image7.png
Greyscale
)
(Kong, Section 3.3, “System state of air combat is defined by the current motion state parameters of UCAV1 and UCAV2, the relative position and angle parameters and future motion state parameters [and the normalization; as provided by the modification of You]. Firstly, the current motion state parameters of the two UCAVs in air combat are considered to form the basic state space 𝒮𝑏𝑎𝑠𝑒:
PNG
media_image12.png
32
507
media_image12.png
Greyscale
”)
You is considered to be analogous to the claimed invention because they are in the same field of reinforcement learning for controlling vehicles in simulations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong and Hu and Vinitsky to incorporate the teachings of You in order to provide concatenation provide faster convergence (You, Section V a.2, “State normalization is an effective means to accelerate the convergence of the DRL algorithm. In the environment of a sparse state space with different scales of state values, the importance of state normalization is self-evident.”) and a mechanism for easier convergence and improved generalization (You, Section IV B., “In practice, for DRL algorithms, we find that it is easier to converge when high-order difference variables are used to describe the environment state than when pure absolute variables are used. For example, the generalization ability of an NN for the relative displacement between two objects (expressed as a 3D vector) is stronger than that for the concatenated vector of two objects' positions (expressed as a six-dimensional vector).”)
In regards to claim 3,
Kong in view of Hu, Vinitsky and You teach The method of claim 2,
You teaches wherein training the machine learning algorithm further comprises: generating a concatenation of the observation of the simulated environment and the normalization;
(You, Section IV B., “The UCAV in Explorer takes sensor feedback from a partially observable environment and stores this feedback within a memory bank. For environmental observation, the feedback signal received by the UCAV at time t can be resolved into a nine-dimensional vector:
PNG
media_image13.png
97
414
media_image13.png
Greyscale
where CON represents a function used to concatenate vectors,
PNG
media_image14.png
22
427
media_image14.png
Greyscale
and
PNG
media_image15.png
23
425
media_image15.png
Greyscale
PNG
media_image16.png
23
43
media_image16.png
Greyscale
Equation 15 indicates that Ot is composed of three
3D vectors: the spatial position of the UCAV,
PNG
media_image17.png
22
117
media_image17.png
Greyscale
the relative displacement between the target and the UCAV,
PNG
media_image18.png
21
112
media_image18.png
Greyscale
and the relative velocity between the target
and the UCAV,
PNG
media_image19.png
26
143
media_image19.png
Greyscale
”)
You also teaches normalization of the values (batch normalization)
(You, Section V A 2), “State normalization is an effective means to accelerate the convergence of the DRL algorithm. In the environment of a sparse state space with different scales of state values, the importance of state normalization is self-evident. In this work, based on Equation 15, we present the following state normalization approach: [generating a concatenation ie a concatenation of the state vectors of the observation of the simulated environment and the normalization]
PNG
media_image20.png
117
467
media_image20.png
Greyscale
where indicates the normalization operator.
Equation 20 shows that by scaling the original state component by the upper bound of the set interval, an ideal normalized state with a value of less than 1 can be generated. In fact, a recent technology called batch normalization [44] in deep learning can also achieve similar results. This technique normalizes each dimension across the samples in a minibatch to have unit mean and variance. We separately used these two methods to test the Explorer game. The results show that the effect of manually ensuring that the state units are within the set range is less time consuming in training than the batch normalization, i.e., the former's convergence rate is faster. This phenomenon may be related to the simplification of the learning process.”)
Thus, Kong in view of Hu with the normalization and concatenation of You teaches and wherein updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the normalization comprises updating the machine learning algorithm based on the reward signal and the concatenation.
(Kong, Algorithm 2 teaches updating the actor (line 20) based on the observation and reward signal (line 9), and the randomly-sampled values wherein the concatenation and normalization would be reflected in the state space wherein the state vectors would be concatenated to make up the state space ie lines 5 and 13
PNG
media_image7.png
488
681
media_image7.png
Greyscale
)
(Kong, Section 3.3, “System state of air combat is defined by the current motion state parameters of UCAV1 and UCAV2, the relative position and angle parameters and future motion state parameters [and the normalization…and the concatenation; as provided by the modification of You]. Firstly, the current motion state parameters of the two UCAVs in air combat are considered to form the basic state space 𝒮𝑏𝑎𝑠𝑒:
PNG
media_image12.png
32
507
media_image12.png
Greyscale
”)
You is considered to be analogous to the claimed invention because they are in the same field of reinforcement learning for controlling vehicles in simulations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong and Hu and Vinitsky to incorporate the teachings of You in order to provide concatenation provide faster convergence (You, Section V a.2, “State normalization is an effective means to accelerate the convergence of the DRL algorithm. In the environment of a sparse state space with different scales of state values, the importance of state normalization is self-evident.”) and a mechanism for easier convergence and improved generalization (You, Section IV B., “In practice, for DRL algorithms, we find that it is easier to converge when high-order difference variables are used to describe the environment state than when pure absolute variables are used. For example, the generalization ability of an NN for the relative displacement between two objects (expressed as a 3D vector) is stronger than that for the concatenated vector of two objects' positions (expressed as a six-dimensional vector).”)
Claim 15 is rejected on the same rationale under 35 U.S.C. 103 as claim 2.
Claim 16 is rejected on the same rationale under 35 U.S.C. 103 as claim 3.
Claim 21 is rejected on the same rationale under 35 U.S.C. 103 as claim 2.
Claim(s) 7-9 are rejected under 35 U.S.C. 103 as being unpatentable over Kong in view of Hu and Vinitsky in further view of U.S. Pat. No. US11188821B1 Kalakrishnan et al. (“Kalakrishnan”).
In regards to claim 7,
Kong and Hu and Vinitsky teaches The method of claim 1,
Kong teaches wherein the machine learning algorithm is associated with one or more weights, wherein training the machine learning algorithm to control the first aircraft
(Kong, Section 5.1.2, “Additionally, the weight of the initialized neural network can be adjusted to be using the Xavier initializer.”)
However, Kong does not explicitly teach comprises training the machine learning algorithm in parallel using a plurality of worker threads, each worker thread configured to utilize the machine learning algorithm during training, and wherein updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the randomly-sampled values comprises:storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in a trajectory vector using a particular worker thread of the plurality of worker threads; sending the trajectory vector from the particular worker thread to a learner thread associated with the plurality of worker threads; updating the one or more weights of the machine learning algorithm based on the trajectory vector using the learner thread; and updating the machine learning algorithm to utilize, using the learner thread, the one or more weights that have been updated.
Kalakrishnan teaches comprises training the machine learning algorithm in parallel using a plurality of worker threads, each worker thread configured to utilize the machine learning algorithm during training,
(Kalakrishnan, Col. 3 lines 1-28, “FIG. 1 shows an example distributed training system 100. The distributed training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.
The distributed training system 100 is a system for distributed training of a global policy neural network and includes multiple local workers 130 a-n, multiple global workers 140 a-n, and a parameter server 190. In particular, the distributed training system 100 trains the global policy neural network to allow the neural network to effectively be used to select actions performed by a robotic agent interacting with a real-world environment.
Each local worker 130 a-n corresponds to one of multiple robotic agents 102 a-n that interacts with a respective instance of the real-world environment 104 a-n [training the machine learning algorithm in parallel using a plurality of worker threads, each worker thread configured to utilize the machine learning algorithm during training; wherein the machine learning algorithm is provided by Kong]. During the training of the global policy neural network, each of the multiple local workers 130 a-n repeatedly performs a local step of an optimization algorithm asynchronously from each of the other local workers 130 a-n. Generally, each local worker operates asynchronously from each other worker because each local worker performs the local step of the optimization algorithm without waiting for other local workers, i.e., each local worker can move on to the next iteration of the local step without waiting for the other local workers to finish the current iteration.”)
Kalakrishnan teaches and wherein updating the machine learning algorithm based on the observation of the simulated environment, the reward signal, and the randomly-sampled values comprises: storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in a trajectory vector using a particular worker thread of the plurality of worker threads;
PNG
media_image21.png
440
379
media_image21.png
Greyscale
(Kalakrishnan, Col. 3 line 61 – Col. 4 line 20, “Each local worker 130 a-n generates a trajectory of state-action pairs by selecting actions to be performed by the instance of the robotic agent corresponding to the local worker while performing the instance of the robotic task. The state-action pairs in the trajectory are data identifying states of the environment that occurred during the instance of the task, and, for each of the states, the action performed by the agent while the environment was in the state. Generating the trajectory is described in more detail below with reference to FIG. 3 …
The local worker 130 a-n then generates an optimized trajectory using the optimized local policy controller 120 a-n and stores the optimized trajectory in a replay memory 150 a-n associated with the local worker 130 a-n. [storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in a trajectory vector using a particular worker thread of the plurality of worker threads; wherein the trajectory vector of Kalakrishnan is substantially similar to the vector stored in the replay buffer of Kong].”)
Kalakrishnan teaches sending the trajectory vector from the particular worker thread to a learner thread associated with the plurality of worker threads;
(Kalakrishnan, Col. 5 lines 7-17, “In particular, in the global step, each global worker 140 a-n samples a stored optimized trajectory from one of the one or more replay memories 150 a-n accessible to the global worker 140 a-n.
Each global worker 140 a-n trains the replica of the global policy neural network 110 a-n maintained by the global worker on the sampled optimized trajectory to determine delta values for the parameters of the global policy neural network [sending the trajectory vector from the particular worker thread to a learner thread associated with the plurality of worker threads], e.g., using supervised learning. Determining delta values is described in more detail below with reference to FIG. 4.”)
Kalakrishnan teaches updating the one or more weights of the machine learning algorithm based on the trajectory vector using the learner thread; and updating the machine learning algorithm to utilize, using the learner thread, the one or more weights that have been updated.
(Kalakrishnan, Col. 5 lines 18-29, “After the delta values for the parameters have been determined, the global worker 140 a-n sends the delta values to the parameter server 190.
The parameter server 190 receives delta values from the global workers 140 a-n, updates current values of the parameters of the global policy neural network using the delta values [updating the one or more weights of the machine learning algorithm based on the trajectory vector using the learner thread; wherein the delta values is based on the trajectory vector and the delta vector is provided by the learner thread], and provides the updated values of the parameters to the global workers for use in training the replicas of the global policy neural network maintained by the global workers [updating the machine learning algorithm to utilize, using the learner thread, the one or more weights that have been updated]. The operations performed by the parameter server 190 during the training of the global policy neural network are described in more detail below with reference to FIG. 5.”)
PNG
media_image22.png
622
705
media_image22.png
Greyscale
Kalakrishnan is considered to be analogous to the claimed invention because they are in the same field of reinforcement learning. Further, Kalakrishnan is reasonably pertinent to the problem the inventor faced in accelerating the learning process. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong in view of Hu and Vinitsky to incorporate the teachings of Kalakrishnan in order to provide a distributed training system to accelerate the training of the network and efficiently use the system’s computational resources (Kalakrishnan, Summary, “The distributed training system as described in this specification provides a solution that breaks dependencies of each worker at a point that effectively leverages the distributed nature of the system to improve the training of the global policy neural network. That is, it is difficult to determine what elements of the system can be accessed, delayed, utilized, etc., by each worker without affecting the performance of the other workers and the overall system as accomplished by the distributed training system described in this specification. Additionally, assumptions of the training algorithm are necessarily violated when applied to this distributed learning process, and the distributed training system as described in this specification effectively trains the global policy neural network despite these violations by effectively dividing the training process across local and global workers. For example, assumptions that the global policy is constantly up-to-date relative to local policies are violated by distributing the training across multiple asynchronous workers. Furthermore, the distributed training system as described in this specification distributes the training in such a way that the benefit of a greater data set size obtained by using multiple local workers outweighs the additional bias introduced by asynchronous training, as may occur with other distributed frameworks. The distributed training system as described in this specification effectively leverages the increased amount of data that can be generated to accelerate the training of a global policy neural network that implements a complex, nonlinear neural network policy. Thus, the distributed training system efficiently makes use of the computational resources allocated to the system to more quickly train the global policy neural network and therefore learn a control policy for a robotic agent.”)
In regards to claim 8,
Kong in view of Hu, Vinitsky and Kalakrishnan teach The method of claim 7,
Kalakrishnan teaches wherein storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in the trajectory vector comprises storing a plurality of observations of the simulated environment, a plurality of reward signals, and a plurality of randomly-sampled values
PNG
media_image21.png
440
379
media_image21.png
Greyscale
(Kalakrishnan, Col. 3 line 61 – Col. 4 line 20, “Each local worker 130 a-n generates a trajectory of state-action pairs by selecting actions to be performed by the instance of the robotic agent corresponding to the local worker while performing the instance of the robotic task. The state-action pairs in the trajectory are data identifying states of the environment that occurred during the instance of the task, and, for each of the states, the action performed by the agent while the environment was in the state. Generating the trajectory is described in more detail below with reference to FIG. 3 …
The local worker 130 a-n then generates an optimized trajectory using the optimized local policy controller 120 a-n and stores the optimized trajectory in a replay memory 150 a-n associated with the local worker 130 a-n. [wherein storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in the trajectory vector comprises storing (1060) a plurality of observations of the simulated environment, a plurality of reward signals, and a plurality of randomly-sampled values; wherein the trajectory vector of Kalakrishnan is substantially similar to the vector stored in the replay buffer of Kong].”)
Kalakrishnan teaches obtained over a plurality of episodes of interactions between the first and second aircraft within the simulated environment in the trajectory vector using the particular worker thread
(Kalakrishnan, Col. 3 lines 16-28, “Each local worker 130 a-n corresponds to one of multiple robotic agents 102 a-n that interacts with a respective instance of the real-world environment 104 a-n. During the training of the global policy neural network, each of the multiple local workers 130 a-n repeatedly performs a local step of an optimization algorithm asynchronously from each of the other local workers 130 a-n [obtained over a plurality of episodes of interactions between the first and second aircraft within the simulated environment in the trajectory vector using the particular worker thread; wherein the instances of the real-world environment is substantially similar to the episodes of interactions between the first and second aircraft within the simulated environment and the iterations is the plurality of episodes that is substantially similar to the iteration of episodes in Algorithm 2 line 6 of Kong]. Generally, each local worker operates asynchronously from each other worker because each local worker performs the local step of the optimization algorithm without waiting for other local workers, i.e., each local worker can move on to the next iteration of the local step without waiting for the other local workers to finish the current iteration.”)
However, Kalakrishnan does not explicitly teach wherein an episode of interactions between the first and second aircraft within the simulated environment of the plurality of episodes of interactions is associated with a predetermined amount of time, and wherein generating the randomly-sampled values comprises generating the randomly-sampled values at a start of each of the plurality of episodes.
Kong teaches wherein an episode of interactions between the first and second aircraft within the simulated environment of the plurality of episodes of interactions is associated with a predetermined amount of time,
(Kong, Algorithm 2 line 6 for max episode length (ie a predetermined amount of time)].
PNG
media_image23.png
81
679
media_image23.png
Greyscale
”)
(Kong, Section 5.1.2, “In a 1-vs.-1 WVR air combat simulation process, the decision period is set to 1 s and the number of maximum step of an episode is set to 150. In an episode simulation, if a UCAV satisfies Equation (8) for five consecutive steps, the episode is terminated, and the UCAV wins the air combat. If there is no win or loss beyond the maximum stride length, then the air combat is considered to be a tie.”)
Kong teaches and wherein generating the randomly-sampled values comprises generating the randomly-sampled values at a start of each of the plurality of episodes.
(Kong, Algorithm 2 lines 4-5 teaches the initialization of the aircrafts at the start of each episode)
In regards to claim 9,
Kong in view of Hu, Vinitsky and Kalakrishnan teach The method of claim 8,
Kalakrishnan teaches wherein storing the plurality of observations of the simulated environment, the plurality of reward signals, and the plurality of randomly-sampled values obtained over the plurality of episodes of interactions between the first and second aircraft within the simulated environment in the trajectory vector using the particular worker thread comprises storing a plurality of observations of the simulated environment, a plurality of reward signals, and a plurality of randomly-sampled values
PNG
media_image21.png
440
379
media_image21.png
Greyscale
(Kalakrishnan, Col. 3 line 61 – Col. 4 line 20, “Each local worker 130 a-n generates a trajectory of state-action pairs by selecting actions to be performed by the instance of the robotic agent corresponding to the local worker while performing the instance of the robotic task. The state-action pairs in the trajectory are data identifying states of the environment that occurred during the instance of the task, and, for each of the states, the action performed by the agent while the environment was in the state. Generating the trajectory is described in more detail below with reference to FIG. 3 …
The local worker 130 a-n then generates an optimized trajectory using the optimized local policy controller 120 a-n and stores the optimized trajectory in a replay memory 150 a-n associated with the local worker 130 a-n. [wherein storing one or more observations of the simulated environment, one or more reward signals, and one or more randomly-sampled values in the trajectory vector comprises storing a plurality of observations of the simulated environment, a plurality of reward signals, and a plurality of randomly-sampled values; wherein the trajectory vector of Kalakrishnan is substantially similar to the vector stored in the replay buffer of Kong].”)
However, Kalakrishnan does not explicitly teach obtained over an epoch of interactions between the first and second aircraft within the simulated environment, and wherein the epoch of interactions between the first and second aircraft within the simulated environment comprises a predetermined number of episodes of interactions between the first and second aircraft within the simulated environment
Kong teaches obtained over an epoch of interactions between the first and second aircraft within the simulated environment, and wherein the epoch of interactions between the first and second aircraft within the simulated environment comprises a predetermined number of episodes of interactions between the first and second aircraft within the simulated environment.
(Kong, Algorithm 2 line 4 and 6 teaches an epoch of interaction ie an episode wherein each episode comprises of a predetermined number of episodes ie t to max-episode-length
PNG
media_image24.png
78
673
media_image24.png
Greyscale
)
(Kong, Section 5.1.2, “In a 1-vs.-1 WVR air combat simulation process, the decision period is set to 1 s and the number of maximum step of an episode is set to 150. In an episode simulation, if a UCAV satisfies Equation (8) for five consecutive steps, the episode is terminated, and the UCAV wins the air combat. If there is no win or loss beyond the maximum stride length, then the air combat is considered to be a tie.”)
Claim(s) 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Kong in view of Hu and Vinitsky in further view of U.S. Pat. No. US10935938B1 Bertram et al. (“Bertram”)
In regards to claim 11,
Kong and Hu and Vinitsky teaches The method of claim 1,
Bertram teaches further comprising: after training the machine learning algorithm, using the trained machine learning algorithm to control a non-simulated aircraft.
(Bertram, Col. 15 line 64-Col. 16 line 9, “In some embodiments, a reinforcement learning sub-system 142 (see FIG. 2) utilizes the gradient and a machine learning algorithm [after training the machine learning algorithm, using the trained machine learning algorithm] such as a neural net to improve the maneuver performance of the maneuver controller 136. For example, the gradient may be an environment with a set of states (e.g., scores) and the reinforcement learning sub-system 142 may be configured to recognize a set of actions (e.g., possible maneuvers) and the reinforcement learning sub-system 142 may learn from one or more actions selected to maximize a reward function (e.g., value-based maximizing function) or based on optimizing scores such that the highest or lowest scores are associated with maneuvers performed by the maneuver controller 136 [to control].”)
(Bertram, Col. 15 lines 20-24, “In an exemplary embodiment, the maneuver controller 136 outputs vehicle control commands to a failsafe vehicle controller for performing the vehicle control commands. For example, the maneuver controller 136 may output one or more flight control commands to an autopilot 138.”)
(Bertram, Col. 5 lines 15-21, “In some embodiments, the system 100 is located on a platform 108. The platform 108 may be any machine or vehicle that uses AI to determine an output based on inferences and observations. For example, the platform 108 may include, but is not limited to, a manned or unmanned aerial vehicle (UAV), an unmanned aircraft system (UAS), and a tactical vehicle.”)
Bertram is considered to be analogous to the claimed invention because they are in the same field of reinforcement learning for aircraft simulation and control. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kong and Hu and Vinitsky to incorporate the teachings of Bertram in order to provide a real-world vehicle and control system to implement the model of Kong as doing so would enable autonomy of a system through constant improvements (Bertram, Abstract, “Machine learning, evaluating, and reinforced learning within systems or apparatuses enables autonomy to a complexity level beyond automation. Inferences are made using machine learning based on observations, images, or video feed of operator input. The inferences are evaluated or classified and maneuvers are performed based on the evaluating or the classification. The performed maneuvers may be further evaluated for scoring or weighting. The reinforcement learning may perform updates based on the scoring, weighting, and a maximizing reward function such that the machine learning is constantly improving.”)
In regards to claim 12,
Kong in view of Hu, Vinitsky and Bertram teach The method of claim 11,
Bertram teaches wherein using the trained machine learning algorithm to control the non-simulated aircraft comprises using the trained machine learning algorithm to control the non-simulated aircraft using one or more control systems of the non-simulated aircraft.
(Bertram, Col. 15 line 64-Col. 16 line 9, “In some embodiments, a reinforcement learning sub-system 142 (see FIG. 2) utilizes the gradient and a machine learning algorithm such as a neural net to improve the maneuver performance of the maneuver controller 136. For example, the gradient may be an environment with a set of states (e.g., scores) and the reinforcement learning sub-system 142 may be configured to recognize a set of actions (e.g., possible maneuvers) and the reinforcement learning sub-system 142 may learn from one or more actions selected to maximize a reward function (e.g., value-based maximizing function) or based on optimizing scores such that the highest or lowest scores are associated with maneuvers performed by the maneuver controller 136 [using one or more control systems of the non-simulated aircraft].”)
Claim(s) 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bertram in view of Kong and Hu and Vinitsky.
In regards to claim 14,
Bertram teaches A computing device, comprising: one or more processors; and data storage storing at least executable instructions that, when executed by the one or more processors, cause the computing device to perform functions comprising:
(Bertram, “In a further aspect, embodiments of the inventive concepts disclosed herein are directed to an apparatus for learning practical autonomy. The apparatus may include a non-transitory memory having instructions thereon and a processor in communication with the non-transitory memory, where the processor may be configured to access the instructions to perform or direct steps of an implicit machine learning algorithm.”)
The rest of the steps are taught by the analogous steps of analogous claim 1.
Kong is considered to be analogous to the claimed invention because they are in the same field of air combat simulation and reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bertram to incorporate the teachings of Kong in order to provide a system to generate highly intelligent air combat strategies in order to enable UCAV to independently complete air combat missions (Kong, Abstract, “With the development of unmanned combat air vehicles (UCAVs) and artificial intelligence (AI), within visual range (WVR) air combat confrontations utilizing intelligent UCAVs are expected to be widely used in future air combats. As controlling highly dynamic and uncertain WVR air combats from the ground stations of the UCAV is not feasible, it is necessary to develop an algorithm that can generate highly intelligent air combat strategies in order to enable UCAV to independently complete air combat missions. In this paper, a 1-vs.-1 WVR air combat strategy generation algorithm is proposed using the multi-agent deep deterministic policy gradient (MADDPG). A 1-vs.-1 WVR air combat is modeled as a two-player zero-sum Markov game (ZSMG). A method for predicting the position of the target is introduced into the model in order to enable the UCAV to predict the target’s actions and position. Moreover, to ensure that the UCAV is not limited by the constraints of the basic fighter maneuver (BFM) library, the action space is considered to be a continuous one. At the same time, a potential-based reward shaping method is proposed in order to improve the efficiency of the air combat strategy generation algorithm. Finally, the efficiency of the air combat strategy generation algorithm and the intelligence level of the resulting strategy is verified through simulation experiments. The results show that an air combat strategy using target position prediction is superior to the one that does not use target position prediction.”)
Hu is considered to be analogous to the claimed invention because they are in the same field of aerial simulation and reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bertram and Kong to incorporate the teachings of Hu in order to provide an environment with parameters representing a real UAV to achieve conditions similar to real-world conditions, particularly with different levels of UAVs from countries around the world. (Hu, Section 5, “To achieve conditions similar to real-world conditions, this study designed a 3D simulation environment based on the parameters of the real UAV. At present, countries around the world have made progress in the research of UAVs and created advanced UAVs.”)
Vinitsky is considered to be analogous to the claimed invention because they are in the same field of multi-agent reinforcement learning. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bertram, Kong and Hu to incorporate the teachings of Vinitsky in order to provide a population-based augmentation in order to yield robustness to dynamic variations under standard parameterizations of the adversary (Vinitsky, Abstract, Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by newadversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across robotics benchmarks that the use of an adversarial population results in a more robust policy that also improves out-of-distribution generalization. Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.”)
Claim 20 is rejected on the same rationale under 35 U.S.C. 103 as claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL: Shen, Macheng, and Jonathan P. How. "Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Games." arXiv preprint arXiv:1909.08735 (2019).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASMINE THAI whose telephone number is (703)756-5904. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.T.T./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights