Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the amendments filed 17 September 2025. Claims 1, 10, 18-22, 25, and 26 are amended. Claims 1-26 are pending and have been examined.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08 October 2025 is being considered by the examiner.
Response to Arguments
Applicant’s arguments, see pages 11-14, filed 17 September 2025, with respect to the rejections of Claims 1-26 under 35 U.S.C. 101 for abstract idea have been fully considered and are persuasive. The rejections of Claims 1-26 under 35 U.S.C. 101 have been withdrawn.
APPLICANT'S ARGUMENT: Applicant appears to argue (page 12, paragraph 2) that "The amended independent claims contain limitations that cannot be performed in the human mind, such as, (1) training of a policy neural network using deep reinforcement learning on different training reservoir models, with a reservoir template providing a common reservoir specification of a specific size for different reservoir models, (2) using the policy neural network to generate a field development plan that includes specific well location for a target reservoir, and (3) outputting digital image(s) of the well locations on a graphical user interface."
Applicant appears to argue (page 12, paragraph 4) that "the amended independent claims contain limitations that provide improvement to the technology of field development optimization for reservoirs. The problem with many existing tools for field development optimization is that they are scenario specific."
Applicant appears to argue (page 13, paragraph 1) that "The limitations of the amended independent claims include improvements to field development optimization that is more flexible than the existing techniques. The limitations of the amended independent claims include improvements to the capabilities of machine-learning models ... to generate field development plans for a class of reservoirs-reservoir with parameter values that fall within a specific range of applicability for deep reinforcement learning."
EXAMINER'S RESPONSE: Examiner agrees. The rejections have been withdrawn in light of arguments and/or amendments.
Applicant's arguments, see pages 14-15, filed 17 September 2025, with respect to the rejections of Claims 1, 2, 4, 5, 7, 8, 17, and 19 under 35 U.S.C. 102(a)(1) have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant appears to argue (page 14, paragraph 3) that "Nasir fails to disclose or suggest use of a reservoir template that provides a common reservoir specification of a specific size for different reservoir models, with the reservoir template enabling flexible application of a policy neural network to generate field development plans for different reservoirs, as recited in amended independent claim 1."
Applicant appears to argue (page 15, paragraph 1) that "Nasir fails to disclose or suggest such use of a reservoir template to train and use a policy neural network that is applicable to different reservoirs. Instead, Nasir describes a scenario specific optimization technique for field development."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are now moot. Claims 1, 2, 4, 5, 7, 8, 17-22, and 24-26 are rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini.
Applicant's arguments, see pages 15-16, filed 17 September 2025, with respect to the rejections of Claims 3, 6-12, 14-16, 18, 20-24, and 26 under 35 U.S.C. 103 have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant appears to argue (page 16, paragraph 1) that "Nasir fails to disclose or suggest limitation of amended independent claims. Other cited references fail to cure the deficiency of Nasir. For at least the reasons presented above, independent claims 1, 19, 21, and 26 are not anticipated or rendered obvious by the cited references. Dependent claims are also not anticipated or rendered obvious at least due to their dependency from independent claim 1, 19, 21, or 26."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are now moot. Claims 6 and 9 are now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of Schulman. Claim 10 is now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of He. Claims 11 and 23 are now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of Tang. Claim 13 is now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of Maucec. Claim 14 is now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of Nelson. Claim 16 is now rejected under 35 U.S.C. 103 as obvious in view of Nasir in view of Zhou in view of Roth in view of Castellini in view of Espeholt.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7, 8, 12, 15, 17, 18-22, and 24-26 are rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini").
Regarding Claim 1, Nasir teaches:
A method (Nasir, p. 2, 4. Methodology: "In this section, the action and observation space are described. Finally, the CNN and PPO DRL algorithm are also discussed") of generating a field development plan for a hydrocarbon field development (Nasir, p. 1, 1. Introduction: "The goal of this work is to investigate the performance of convolutional neural network-based (CNN) deep reinforcement learning (DRL) algorithm to the field development optimization problem (with fixed well operational settings)" and "we have the field development optimization (FDO) problem where decisions on the number of wells, their type (extraction/injection), location, drilling sequence, and well operational settings need to be made to optimally extract hydrocarbon in order to maximize an economic metric," where Nasir's optimal field development solution corresponds to the instant field development plan), the method comprising:
obtaining a reservoir template, the reservoir template providing a common reservoir specification (Nasir, 3. Optimization Problem and Data: "Our goal in this work is to use DRL to find a policy (
π
) that maps from states (representation of the environment/reservoir model) to optimal actions/decisions for the field development problem," where Nasir's state representation of the reservoir model corresponds to the instant reservoir template, where Nasir's state space comprises four 2D maps, as depicted at p. 3, Figure 1. The four 2D maps that make op the map component of the state space, (a) Permeability map (in log scale), (b) Pressure map, (c) Saturation map, (d) Well location map) of a specific size (Nasir, p. 3, 4.1. Action and state space: "
N
x
and
N
y
represents the number of grid blocks in the areal
x
and
y
directions. The reservoir model in this work has
N
x
=
N
y
=
60
. Fig. 1 shows a typical representation of the state space maps at a particular stage of the development") for different reservoir models (Nasir, p. 3, 4.1. Action and state space: "The 2D maps in the state space include the permeability, pressure, saturation and location maps. The permeability map is a static (in this work) measure that indicates the conductivity conductivity of fluid in different section of the subsurface model. The dynamic pressure and saturation maps are generated by the reservoir simulator. They depend on the action taken by the agent," where Nasir's dynamically changing state corresponds to the instant different models), wherein input channels of the reservoir template represent geological properties, rock-fluid properties, operational constraints, economic conditions, or any combination thereof (Nasir, p. 3, 4.1. Action and state space: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. The 2D maps in the state space include the permeability, pressure, saturation and location maps," where Nasir's permeability, pressure, saturation and location correspond to the instant geological properties, rock-fluid properties, and operational constraints, and where Nasir's combination thereof is the combination not including economic conditions) to define the different reservoir models for deep reinforcement learning (Nasir, p. 2, 3. Optimization Problem and Data: "AD-GPRS ... is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space. ... This entails saving information of previous flow simulations (due to actions taken in previous drilling stages) that allows for the simulation to be restarted at the beginning of the next drilling stage without the need to run the flow simulation from scratch. In this work, five drilling stages are considered each of length 150 days");
generating a plurality of training reservoir models (Nasir, 4.2. DRL algorithm and CNN architecture: "A sample size of 320 episodes (full flow simulation) is specified to collect experiences by rolling out the policy until the terminal state is reached.... Adam optimizer is used with an initial learning rate of 0.001 and a batch size of 160 episodes is used for training" where Nasir's training samples were generated by simulation, as in p. 2, 3. Optimization Problem and Data: "In this work, the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space") of varying values of input the channels of the reservoir template (Nasir, p. 3, 4.1. Action and state space: "The permeability and pressure map are normalized to values between 0 and 1. The saturation which is a percentage is given between 0 and 1 and hence do not need to be normalized. The well location map however is more like a mask with zeros and have value of -1 where a producer is drilled and 1 where an injector is drilled. ... The spatial component of the state space
s
m
∈
R
N
y
×
N
x
×
c
, where
c
defines the number of 2D maps (
c
=
4
in this work)"), the varying values of the input channels falling within a specified range of applicability for the deep reinforcement learning (Nasir, p. 3, 4.1. Action and state space: "The reinforcement learning agent performs certain actions in the environment. This results in a new state for the environment and a reward signal that indicates the quality of the action. The action and observation spaces for the FDO problem are now described. ... ¶ At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. ... The dynamic pressure and saturation maps are generated by the reservoir simulator. They depend on the action taken by the agent");
normalizing the varying values of the input channels from the plurality of training reservoir models to generate normalized values of the input channels (Nasir, p. 3, 4.1. Action and state space: "The dynamic pressure and saturation maps are generated by the reservoir simulator. They depend on the action taken by the agent. ... ¶ The permeability and pressure map are normalized to values between 0 and 1");
constructing a policy neural network and a value neural network (Nasir, p. 4, Fig. 2, depicting CNN architectures taking state as input and producing policy and value results, and p. 4, 4.2. DRL algorithm and CNN architecture: "The individual concatenated tensors are further processed by fully connected layers to produce the probability distribution of the action space condition on the state space (
π
a
s
for the policy network) and a single digit that indicated the value of the state (
V
s
for the value network)") that project a state of a reservoir represented by the normalized values of the input channels (Nasir, p. 4, Fig. 2, depicting the networks taking "spatial component of state space state" as input, and p. 3: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. The 2D maps in the state space include the permeability, pressure, saturation and location maps") to a field development action for the reservoir (Nasir, p. 3, 4.1. Action and state space: "The reinforcement learning agent performs certain actions in the environment. This results in a new state for the environment and a reward signal that indicates the quality of the action" and p. 4, 4.2. DRL algorithm and CNN architecture: "The policy update for PPO-clip using multiple steps of stochastic gradient descent is given by:
θ
k
+
1
=
arg
max
θ
E
s
,
a
∼
π
θ
k
L
s
,
a
,
θ
k
,
θ
(3)
," where
a
corresponds to action) and a value of the state of the reservoir respectively (Nasir, p. 4, 4.2. DRL algorithm and CNN architecture: "The individual concatenated tensors are further processed by fully connected layers to produce ... a single digit that indicated the value of the state (
V
s
for the value network)"); and
training the policy neural network and the value neural network (Nasir, p. 5, 4.2. DRL algorithm and CNN architecture: "A single Nvidia V100 GPU is used to train the CNN network" where Nasir's CNN network comprises policy and value networks, as in p. 4, 4.2. DRL algorithm and CNN architecture: "CNN architectures ... shown in Fig. 2, are considered. ... The output of the third layer is shared by the policy and value arms of the network") using the deep reinforcement learning (Nasir, p. 1, 1. Introduction: "The goal of this work is to investigate the performance of convolutional neural network-based (CNN) deep reinforcement learning (DRL) algorithm to the field development optimization problem") on the plurality of training reservoir models (Nasir, p. 4, 4.2. DRL algorithm and CNN architecture: "A sample size of 320 episodes (full flow simulation) is specified to collect experiences by rolling out the policy until the terminal state is reached using 40 CPU processors. ... Adam optimizer is used with an initial learning rate of 0.001 and a batch size of 160 episodes is used for training," where Nasir's non-terminal states correspond to the instant models) with a reservoir simulator as an environment (Nasir, p. 2, 3. Optimization Problem and Data: "the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space") wherein the reservoir simulator solves ... to evolve the state over time (Nasir, p. 1, 1. Introduction: "If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made. This is a very challenging mixed-integer nonlinear programming (MINLP) problem that requires a significant number of expensive flow simulations" and p. 2, 3. Optimization Problem and Data: "the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space. ... In this work, five drilling stages are considered each of length 150 days") the policy neural network generating a field development plan comprising well counts, well locations, well type, well sequence, or any combination thereof (Nasir, p. 2, 3. Optimization Problem and Data: "Our goal in this work is to use DRL to find a policy (
π
) that maps from states (representation of the environment/reservoir model) to optimal actions/decisions for the field development problem" where p. 1, 1. Introduction: "In the field development problem, the production life-cycle is divided into a number of discrete drilling stages. Our goal in each stage is to decide if to drill a well or not. If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made") to improve profitability of a hydrocarbon field development (Nasir, p. 2, 3. Optimization Problem and Data: "In this work, the cumulative reward
J
is defined as the net present value (NPV)" where p. 1, Abstract: "The field development optimization (FDO) problem represents a ... problem in which we seek to obtain the number of wells, their type, location, and drilling sequence that maximizes an economic metric"), wherein the policy neural network is trained (Nasir, p. 2, 3. Optimization Problem and Data: "the DRL problem can be represented as follows:
max
θ
J
π
a
s
,
θ
(1)
where
J
is the cumulative reward/objective function to be optimized," where
π
a
s
,
θ
is Nasir's policy network) to generate the field development plan (Nasir, p. 1, 1. Introduction: "In the field development problem, the production life-cycle is divided into a number of discrete drilling stages. Our goal in each stage is to decide if to drill a well or not. If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made" and p. 5, 5. Results: "The results for the optimization problem using the small and large CNN architectures are now presented. Fig. 3 shows the evolution of the cumulative reward with number of iterations. ... ¶ ... Although, the DRL-based solutions obtained similar number of wells, the solution of the small network entails a development with three producers and one injector, while the large network's solution includes two injectors and two producers. In both solutions, however, an injector is drilled in the first stage") for a given reservoir having given values of the input channels that fall with the specified range of applicability for the deep reinforcement learning (Nasir, p. 2, 3. Optimization Problem and Data: "In this work, the cumulative reward
J
is defined as the net present value (NPV). Following [19], we compute NPV as follows:
PNG
media_image1.png
181
598
media_image1.png
Greyscale
Here
N
i
and
N
p
are the number of injection and production wells, respectively,
N
t
is the number of time steps in the flow simulation,
t
k
and
Δ
t
k
are the time and time step size at time step
k
,
t
i
is the time at which well
i
is drilled.... The rates of oil/water production and water injection, for well
i
at time step
k
are, respectively,
q
o
,
k
i
,
q
p
w
,
k
i
, and
q
i
w
,
k
i
. Similar equation can be written for a specific drilling stage by considering only the production, injection and drilling information at that stage");
obtaining values for the input channels of the reservoir template for a target reservoir, the obtained values for the input channels falling within the specific range of applicability (Nasir, p. 3, 4.1. Action and state space: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. ... The dynamic pressure and saturation maps are generated by the reservoir simulator");
rescaling and normalizing the obtained values for the input channels to generate rescaled and normalized target input values, wherein the obtained values are rescaled to scale of the reservoir template (Nasir, p. 3, 4.1. Action and state space: "The permeability and pressure map are normalized to values between 0 and 1. The saturation which is a percentage is given between 0 and 1 and hence do not need to be normalized," which reasonably suggests that Nasir's obtained permeability/pressure values are non normalized, and where [0,1] normalization is also rescaling under BRI);
generating a given field development plan for the target reservoir (Nasir, p. 6, Policy generalization: "The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells (except the slight alteration of the location of the producer drilled in the third drilling stage), while drilling an injector in drilling stage two as will be expected") on the reservoir template with the rescaled and normalized target input values (Nasir, p. 3, 4.1. Action and state space: "The dynamic pressure and saturation maps are generated by the reservoir simulator"), the trained policy neural network (Nasir, p. 6, 5. Results: "The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells"), and the reservoir simulator (Nasir, p. 3, 4.1. Action and state space: "The dynamic pressure and saturation maps are generated by the reservoir simulator");
generate a final field development plan for the target reservoir, wherein the final field development plan is mapped to the target reservoir, the final field development plan comprising well locations in the target reservoir (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action," depicting well locations 1-4 of the optimal/final plan, and p. 6, Policy generalization: "The action proposed by the policy of the large network in the second drilling stage, which is to drill a producer (in the northwestern region) is executed in the first stage. ... Interestingly, in the PSO-MADS solution the first action involves a producer drilled in approximately same area in the northwestern region. After choosing this first action, the policy of the large network is followed from the second to fifth stage. The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells (except the slight alteration of the location of the producer drilled in the third drilling stage), while drilling an injector in drilling stage two as will be expected," where Nasir's reservoir with a northwest region corresponds to the instant target reservoir);
outputting ... at least a portion of the final field development plan, the portion of the final field development plan ... including one or more digital images of the well locations in the target reservoir (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action," depicting well locations 1-4 of the optimal/final plan).
Nasir teaches training the policy neural network and the value neural network using the deep reinforcement learning on the plurality of training reservoir models with a reservoir simulator as an environment.
Nasir does not explicitly teach wherein the reservoir simulator solves a set of governing equations.
However, Zhou teaches:
wherein the reservoir simulator solves a set of governing equations (Zhou, p. 51, 3.3.1 First level: global matrix: "In general-purpose reservoir simulation, we often need to solve a coupled system that contains many complex objects, such as the reservoir model, wells, and surface facilities. Each of these objects has its own set of nonlinear equations and variables. ... [I]n fully coupled schemes, a global Jacobian matrix that incorporates all the derivatives of the different governing equations with respect to all the variables, is required" and p. 87, Second stage: overall system: "The reservoir and facility models are generally strongly coupled via pressure. Due to the pressure decoupling at the first stage of CPR, we may solve the overall system locally (i.e., one submatrix at a time) in the second stage").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Nasir regarding training the policy neural network and the value neural network using the deep reinforcement learning on the plurality of training reservoir models with a reservoir simulator as an environment with those of Zhou regarding wherein the reservoir simulator solves a set of governing equations.
The motivation to do so would be to facilitate modeling and research of complex subsurface processes computationally (Zhou, p. 3, 1.1 Background and Dissertation Outline: "we need to think about all the unresolved and upcoming challenges associated with the development of a general-purpose reservoir-simulation platform. The research platform should be able to accommodate the growing variety and complexity of subsurface nonlinear processes that must be modeled accurately and efficiently. ¶ ... [O]ur objective is to establish a general-purpose numerical simulation framework that can be used as a flexible, extensible, and computationally efficient platform for reservoir simulation research. ... Given the discrete form of the governing nonlinear residual equations and declaration of the independent variables, the AD library employs advanced expression templates with block data-structures to automatically generate compact computer code for the Jacobian matrix").
The Nasir/Zhou combination may not explicitly teach rescaling the given field development plan to scale of the target reservoir.
However, Roth teaches:
rescaling the given field development plan to scale of the target reservoir (Roth, [0254]: "In step (23), the 'merged data' data table is provided to a prediction algorithm that predicts well production for a variety of production intervals (e.g. 30-day, 60-day, 90-day) and production types (e.g. oil, gas, water, condensate) at different points in the well's lifecycle ( e.g. permitted, drilled, completed, flowback tested, producing, recompleted)" where Roth's field plans had been normalized previously to 30-day increments, as in [0172]: "production between wells cannot be properly compared without first normalizing the production, which in tum requires a reliable measurement of the Days On. In embodiments of the present disclosure, the production is normalized and interpolated so that 30-day increments of producing days, or Days On, can be compared between wells").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou combination regarding generating a field development plan for the target reservoir on the reservoir template with the rescaled and normalized target input values, the trained policy network, and the reservoir simulator with those of Roth regarding rescaling the generated field plan to scale of the target reservoir model to generate a final field plan for the target reservoir.
The motivation to do so would be to ensure that actual and forecasted production plans provide consistent information (Roth, [0196]: "FIG. 24 is a production data table showing normalized production attributes obtained both with and without use of the 'days on' estimation technique.... As evident from the table, the lower volumes in the non-corrected data of columns 744 and 748 could lead to wells being falsely considered as poor producers, when in fact they are not").
The Nasir/Zhou/Roth combination may not explicitly teach outputting, on a graphical user interface, at least a portion of the final field development plan.
However, Castellini teaches:
outputting, on a graphical user interface, at least a portion of the final field development plan (Castellini, [0077]: "FIG. 12 illustrates system 200 that can be used to perform the iterative response Surface methods. Such as computer-implemented method 100, previously described herein. System 200 includes user interface 210, such that an operator can actively input information and review operations of system 200. User interface 210 can be any means in which a person is capable of interacting with system 200 Such as a keyboard, mouse, touch-screen display, or Voice-command controls").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth combination regarding generating a final field development plan with the teachings of Castellini regarding outputting at least a portion of the final field development plan on a graphical user interface.
The motivation to do so would be to facilitate performance optimization or decision making by users (Castellini, [0084]: "A visual display can be produced, such as through reporting unit 250 or user interface 210. ... The displayed information can be utilized to forecast or optimize the production performance of the sub terranean reservoir, or used in making other reservoir management decisions").
Regarding Claim 19, Nasir teaches:
a system of generating a field development plan for a hydrocarbon field development, the system comprising: one or more physical processors configured by machine-readable instructions (Nasir, p. 3, 4.2. DRL algorithm and CNN architecture: "A single Nvidia V100 GPU is used to train the CNN network. The RLlib package [10] handles the communication between the GPU and CPUs and the parallel execution of the flow simulation with the CPUs using a custom field development gym environment developed in this work," where Nasir's use of the RLib package corresponds to the instant instructions) to execute precisely those steps recited in the rejection of Claim 1.
Regarding Claim 2, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein at least one two dimensional (2D) digital image is utilized to represent the values after normalization of each input channel (Nasir, p. 6, Figure 5. "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action," where Nasir's depicted optimal solution represents the instant varying values of input channels after normalization).
Regarding Claim 3, the rejection of Claim 1 is incorporated.
Castellini further teaches:
wherein at least one three dimensional (3D) digital cube is utilized (Castellini, Fig. 3, which appears to depict 3D cube representing oil field input data, and [0015]: "FIG. 3 is a schematic view of a response surface for cumulative field oil production") to represent the values ... of each input channel (Castellini, [0058]: "FIG. 3 shows a response surface of the cumulative field oil production... The response Surface is drawn as a function of the permeability multipliers in the X and y direction, which largely influence the field production. Importantly, this design problem is 5-dimensional, and the projection in the 2D
k
x
,
k
y
plane is considered. The concentration of sampling points in the lower-left corner of the
k
x
,
k
y
, space, as discussed in FIG.2, appear exaggerated by the projection as these points differ in the other remaining three dimensions" where Castellini's five input channels are listed at [0056]: "The construction of iterative response surfaces was applied to a newly offshore sand reservoir. The objective of this example was to obtain a probability distribution function for the cumulative oil production of the field after 10 years in production as a function of five uncertain geological parameters. The geological model for this field included 1,051.200 cells. The production forecasts were subject to five major Sources of uncertainty: connate water saturation, rock compressibility, and permeability multipliers in the x, y and z directions") after normalization (Castellini, [0058]: "FIG. 3 shows a response surface of the cumulative field oil production normalized to a reference value of 1 for this example").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding normalizing the varying values of the input channels to generate normalized values of the input channels with the further teachings of Castellini regarding wherein at least one three dimensional (3D) digital cube is utilized to represent the values after normalization of each input channel.
The motivation to do so would be to provide a representative surface useful for generating predictive samples based on the input data (Castellini, [0059]: "After two iterations and a total of 36 runs ... the iterative response surface method reaches a predetermined precision threshold. ... The final proxy thus offers a predictive response surface for cumulative oil production as a function of the five input parameters. The final proxy can be sampled with a Monte-Carlo sampling technique to obtain the probability density function of the cumulative oil produced after 10 years in production").
Regarding Claim 4, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein at least portions of the policy neural network and the value neural network comprise convolution layers and residual blocks (Nasir, Figure 2. "Small and large CNN architectures. The first number in the convolutional layers represent the number of filters and the number in parenthesis of the fully connected layers denote the number of neurons. All convolution are of stride one with no padding (except in residual blocks with same padding)").
Regarding Claim 5, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the deep reinforcement learning comprises proximal policy optimization (PPO), Importance weighted Actor-Learner Architecture (IMPALA), or any combination thereof (Nasir, p. 2, 2. Related Work: "The focus in this [i.e., Nasir's] work is on the more challenging field development optimization problem in which the decision to be made include the number of wells, their locations, types and drilling sequence. A CNN-based PPO is considered in order to retain the spatial information present in the observation space (dynamic state and static maps)," where Nasir's combination comprises PPO alone).
Regarding Claim 7, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising using a stochastic gradient descent (SGD) algorithm during the training (Nasir, p. 4, 4.2. DRL algorithm and CNN architecture: "As noted earlier, in this work the proximal policy optimization (PPO) [18] deep reinforcement learning algorithm is considered. ... The PPO with clipped surrogate objective (PPO-clip) is used in this work. The policy update for PPO-clip using multiple steps of stochastic gradient descent is given by: [Eq. 3] where L is given by [Eq. 4]").
Regarding Claim 8, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the policy neural network and the value neural network share weights in at least one layer (Nasir, Figure 2(b), Large network, which depicts the policy and value networks sharing two convolutional layers, three residual blocks, and two fully-connected layers; and p. 4, 4.2. DRL algorithm and CNN architecture: "The large network shown in Fig. 2(b) is similar to the small network except three residual blocks are introduced to replace the last convolution in the shared layers by the policy and value arms").
Regarding Claim 12, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising modifying a value of porosity, a value of transmissibility, or any combination thereof (Nasir, p. 3: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. The 2D maps in the state space include the permeability, pressure, saturation and location maps. ... The dynamic pressure and saturation maps are generated by the reservoir simulator," where Nasir's dynamic permeability corresponds to the instant modified transmissibility).
Castellini further teaches:
a value of porosity, a value of transmissibility, or any combination thereof to represent a fault (Castellini, [0045]: "The sample can be for other properties in addition to permeability. ... In some embodiments, the uncertain properties for sampling are local field properties. Examples of local field properties include the permeability in the neighborhood of a well and the transmissibility of a fault").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding modifying a value of porosity, a value of transmissibility, or any combination thereof with the further teachings of Castellini regarding a value of porosity, a value of transmissibility, or any combination thereof to represent a fault.
The motivation to do so would be to facilitate greater control over modeling of uncertainty and outcome precision (Castellini, [0045]: "The choice of properties to sample depends on the uncertainties for a particular application, and the associated goals for these uncertainties. The choice of properties to sample is not typically dependent on the chosen sampling or experimental design technique. However, the choice of a design can often affect the number of uncertainties and the desired precision of outcomes").
Regarding Claim 15, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the field development action comprises drilling a ... well by location (Nasir, p. 2, 2. Related Work: "The focus in this work is on the more challenging field development optimization problem in which the decision to be made include the number of wells, their locations, types and drilling sequence").
Roth further teaches:
drilling a horizontal well by location of its middle point, angle, and length (Roth, [0180]: "FIGS. 17 A and 17B show side (elevation) and top (plan) views of a well 440, respectively. ... The total measured depth 420 is the total length along the wellbore between the surface location 404 and the bottom hole location 436. ... The midpoint location 432 is the halfway point in measured depth along the horizontal section length 444. The horizontal azimuth 452 is the angle of the horizontal section relative to North").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding the field development action comprising drilling a well by location with the further teachings of Roth regarding drilling a horizontal well by location of its middle point, angle, and length.
The motivation to do so would be to ensure that well production estimates properly account for neighboring wells (Roth, [0182]: "The process of estimating directional survey data depends upon the amount and type of available information" and [0183]: "In some instances, a well may have at least one neighbor with a reported directional survey. For a second well to be considered a neighbor of the first well, the azimuth of a straight line connecting the surface location 404 and bottom hole location 436 of the first well must be within a specified tolerance, and the midpoint of the second well and the straight line connecting the surface location 404 and the bottom hole location 436 of the first well must be within a specified distance. A user or operator of the device 100 may determine the specified tolerance and the specified distance to be used").
Regarding Claim 17, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein at least one input channel of the reservoir template represents a plurality of properties (Nasir, p. 3, 4.1. Action and state space: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. ... The vector component of the state space at each drilling stage
s
v
∈
R
3
contains the normalized (between zero and one) drilling stage number, number of producers and injectors that have been drilled at that stage," where Nasir's observation vector corresponds to the instant input channel representing a plurality of properties).
Regarding Claim 18, the rejection of Claim 1 is incorporated.
Castellini further teaches:
wherein the plurality of training reservoir models comprise multiple three dimensional (3D) reservoir models (Castellini, [0044]: "Referring to FIG. 1, a three-dimensional view of an exemplary reservoir 11 is shown, which has been sampled in accordance with an embodiment of the present invention. Reservoir 11 can be any type of subsurface formation in which hydrocarbons are stored. Such as limestone, dolomite, oil shale, gas shale, sandstone, or a combination thereof. Reservoir 11 is shown with a plurality of production wells 13 and injector wells 15 that are in fluid communication with a hydrocarbon producing Zone of reservoir 11").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding generating a plurality of training reservoir models of varying values of the input channels of the reservoir template with the further teachings of Castellini regarding wherein the plurality of training reservoir models comprise multiple three dimensional (3D) reservoir models.
The motivation to do so would be to facilitate representation and use of global properties of a field (Castellini, [0044]: "the iterative experimental points in this embodiment are not drawn on a field view because the samples are taken such that they represent global field properties. For example, these samples can be properties such as permeability distributions, where each 'sample' represents the permeability distribution of a different full field. Accordingly, each sample can represent a full three-dimensional field with several differing properties from the other samples").
Claims 20 and 22 recites a system configured to execute precisely those steps recited in the rejection of Claim 18.
Regarding Claim 21, Nasir teaches:
A method (Nasir, p. 2, 4. Methodology: "In this section, the action and observation space are described. Finally, the CNN and PPO DRL algorithm are also discussed") of generating a field development plan for a hydrocarbon field development (Nasir, p. 1, 1. Introduction: "The goal of this work is to investigate the performance of convolutional neural network-based (CNN) deep reinforcement learning (DRL) algorithm to the field development optimization problem (with fixed well operational settings)" and "we have the field development optimization (FDO) problem where decisions on the number of wells, their type (extraction/injection), location, drilling sequence, and well operational settings need to be made to optimally extract hydrocarbon in order to maximize an economic metric," where Nasir's optimal field development solution corresponds to the instant field development plan), the method comprising:
obtaining values for input channels according to a reservoir template for a target reservoir (Nasir, p. 3, 4.1. Action and state space: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. ... The dynamic pressure and saturation maps are generated by the reservoir simulator"), the reservoir template providing a common reservoir specification (Nasir, 3. Optimization Problem and Data: "Our goal in this work is to use DRL to find a policy (
π
) that maps from states (representation of the environment/reservoir model) to optimal actions/decisions for the field development problem," where Nasir's state representation of the reservoir model corresponds to the instant reservoir template, where Nasir's state space comprises four 2D maps, as depicted at p. 3, Figure 1. The four 2D maps that make op the map component of the state space, (a) Permeability map (in log scale), (b) Pressure map, (c) Saturation map, (d) Well location map) of a specific size (Nasir, p. 3, 4.1. Action and state space: "
N
x
and
N
y
represents the number of grid blocks in the areal
x
and
y
directions. The reservoir model in this work has
N
x
=
N
y
=
60
. Fig. 1 shows a typical representation of the state space maps at a particular stage of the development") for different reservoir models (Nasir, p. 3, 4.1. Action and state space: "The 2D maps in the state space include the permeability, pressure, saturation and location maps. The permeability map is a static (in this work) measure that indicates the conductivity conductivity of fluid in different section of the subsurface model. The dynamic pressure and saturation maps are generated by the reservoir simulator. They depend on the action taken by the agent," where Nasir's dynamically changing state corresponds to the instant different models), wherein the input channels represent geological properties, rock-fluid properties, operational constraints, economic conditions, or any combination thereof (Nasir, p. 3, 4.1. Action and state space: "At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. The 2D maps in the state space include the permeability, pressure, saturation and location maps," where Nasir's permeability, pressure, saturation and location correspond to the instant geological properties, rock-fluid properties, and operational constraints, and where Nasir's combination thereof is the combination not including economic conditions);
obtaining a policy network, the policy network trained using deep reinforcement learning
(Nasir, p. 2, 3. Optimization Problem and Data: "Our goal in this work is to use DRL to find a policy (
π
) that maps from states (representation of the environment/reservoir model) to optimal actions/decisions for the field development problem" and p. 5, 4.2. DRL algorithm and CNN architecture: "A single Nvidia V100 GPU is used to train the CNN network" where Nasir's CNN network comprises policy and value networks) on a plurality of training reservoir models (Nasir, p. 4, 4.2. DRL algorithm and CNN architecture: "A sample size of 320 episodes (full flow simulation) is specified to collect experiences by rolling out the policy until the terminal state is reached using 40 CPU processors. ... Adam optimizer is used with an initial learning rate of 0.001 and a batch size of 160 episodes is used for training," where Nasir's non-terminal states correspond to the instant models) with a reservoir simulator as an environment (Nasir, p. 2, 3. Optimization Problem and Data: "the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space"), wherein the reservoir simulator solves ... to evolve the state over time (Nasir, p. 1, 1. Introduction: "If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made. This is a very challenging mixed-integer nonlinear programming (MINLP) problem that requires a significant number of expensive flow simulations" and p. 2, 3. Optimization Problem and Data: "the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space. ... In this work, five drilling stages are considered each of length 150 days"), the plurality of training reservoir models defined by varying values of the input channels of the reservoir template (Nasir, 4.2. DRL algorithm and CNN architecture: "A sample size of 320 episodes (full flow simulation) is specified to collect experiences by rolling out the policy until the terminal state is reached," where Nasir's training samples were generated by simulation, as in p. 2, 3. Optimization Problem and Data: "In this work, the Stanford’s Automatic Differentiation-based General Purpose Simulator (AD-GPRS) [20] is used for the flow simulation to obtain quantities from which the objective function is computed and also the dynamic state maps (described later) that are components of the state space"), the varying values of the input channels falling within a specified range of applicability for the deep reinforcement learning (Nasir, p. 3, 4.1. Action and state space: "The reinforcement learning agent performs certain actions in the environment. This results in a new state for the environment and a reward signal that indicates the quality of the action. The action and observation spaces for the FDO problem are now described. ... ¶ At each drilling stage, the state/observation space is represented by both two-dimensional maps and a vector. ... The dynamic pressure and saturation maps are generated by the reservoir simulator. They depend on the action taken by the agent"), wherein the policy neural network is trained (Nasir, p. 2, 3. Optimization Problem and Data: "the DRL problem can be represented as follows:
max
θ
J
π
a
s
,
θ
(1)
where
J
is the cumulative reward/objective function to be optimized," where
π
a
s
,
θ
is Nasir's policy network) to generate the field development plan (Nasir, p. 1, 1. Introduction: "In the field development problem, the production life-cycle is divided into a number of discrete drilling stages. Our goal in each stage is to decide if to drill a well or not. If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made" and p. 5, 5. Results: "The results for the optimization problem using the small and large CNN architectures are now presented. Fig. 3 shows the evolution of the cumulative reward with number of iterations. ... ¶ ... Although, the DRL-based solutions obtained similar number of wells, the solution of the small network entails a development with three producers and one injector, while the large network's solution includes two injectors and two producers. In both solutions, however, an injector is drilled in the first stage") for a given reservoir having given values of the input channels that fall with the specified range of applicability for the deep reinforcement learning (Nasir, p. 2, 3. Optimization Problem and Data: "In this work, the cumulative reward
J
is defined as the net present value (NPV). Following [19], we compute NPV as follows:
PNG
media_image1.png
181
598
media_image1.png
Greyscale
Here
N
i
and
N
p
are the number of injection and production wells, respectively,
N
t
is the number of time steps in the flow simulation,
t
k
and
Δ
t
k
are the time and time step size at time step
k
,
t
i
is the time at which well
i
is drilled.... The rates of oil/water production and water injection, for well
i
at time step
k
are, respectively,
q
o
,
k
i
,
q
p
w
,
k
i
, and
q
i
w
,
k
i
. Similar equation can be written for a specific drilling stage by considering only the production, injection and drilling information at that stage");
rescaling and normalizing the obtained values for the input channels to generate rescaled and normalized target input values, wherein the obtained values are rescaled to scale of the reservoir template (Nasir, p. 3, 4.1. Action and state space: "The permeability and pressure map are normalized to values between 0 and 1. The saturation which is a percentage is given between 0 and 1 and hence do not need to be normalized," which reasonably suggests that Nasir's obtained permeability/pressure values are non normalized, and where [0,1] normalization is also rescaling under BRI);
generating a given field development plan for the target reservoir (Nasir, p. 6, Policy generalization: "The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells (except the slight alteration of the location of the producer drilled in the third drilling stage), while drilling an injector in drilling stage two as will be expected") on the reservoir template with the rescaled and normalized target input values (Nasir, p. 3, 4.1. Action and state space: "The dynamic pressure and saturation maps are generated by the reservoir simulator"), the policy network (Nasir, p. 6, 5. Results: "The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells"), and the reservoir simulator (Nasir, p. 3, 4.1. Action and state space: "The dynamic pressure and saturation maps are generated by the reservoir simulator");
generate a final field development plan for the target reservoir, wherein the final field development plan is mapped to the target reservoir, the final field development plan comprising well locations in the target reservoir (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action," depicting well locations 1-4 of the optimal/final plan, and p. 6, Policy generalization: "The action proposed by the policy of the large network in the second drilling stage, which is to drill a producer (in the northwestern region) is executed in the first stage. ... Interestingly, in the PSO-MADS solution the first action involves a producer drilled in approximately same area in the northwestern region. After choosing this first action, the policy of the large network is followed from the second to fifth stage. The result obtained is shown in Fig. 5 shows the solution obtained with an NPV of $269.06 million. The policy maintained the location of the subsequent wells (except the slight alteration of the location of the producer drilled in the third drilling stage), while drilling an injector in drilling stage two as will be expected," where Nasir's reservoir with a northwest region corresponds to the instant target reservoir);
outputting ... at least a portion of the final field development plan, the portion of the final field development plan ... including one or more digital images of the well locations in the target reservoir (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action," depicting well locations 1-4 of the optimal/final plan).
Nasir teaches training the policy neural network and the value neural network using the deep reinforcement learning on the plurality of training reservoir models with a reservoir simulator as an environment.
Nasir does not explicitly teach wherein the reservoir simulator solves a set of governing equations.
However, Zhou teaches:
wherein the reservoir simulator solves a set of governing equations (Zhou, p. 51, 3.3.1 First level: global matrix: "In general-purpose reservoir simulation, we often need to solve a coupled system that contains many complex objects, such as the reservoir model, wells, and surface facilities. Each of these objects has its own set of nonlinear equations and variables. ... [I]n fully coupled schemes, a global Jacobian matrix that incorporates all the derivatives of the different governing equations with respect to all the variables, is required" and p. 87, Second stage: overall system: "The reservoir and facility models are generally strongly coupled via pressure. Due to the pressure decoupling at the first stage of CPR, we may solve the overall system locally (i.e., one submatrix at a time) in the second stage").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Nasir regarding training the policy neural network and the value neural network using the deep reinforcement learning on the plurality of training reservoir models with a reservoir simulator as an environment with those of Zhou regarding wherein the reservoir simulator solves a set of governing equations.
The motivation to do so would be to facilitate modeling and research of complex subsurface processes computationally (Zhou, p. 3, 1.1 Background and Dissertation Outline: "we need to think about all the unresolved and upcoming challenges associated with the development of a general-purpose reservoir-simulation platform. The research platform should be able to accommodate the growing variety and complexity of subsurface nonlinear processes that must be modeled accurately and efficiently. ¶ ... [O]ur objective is to establish a general-purpose numerical simulation framework that can be used as a flexible, extensible, and computationally efficient platform for reservoir simulation research. ... Given the discrete form of the governing nonlinear residual equations and declaration of the independent variables, the AD library employs advanced expression templates with block data-structures to automatically generate compact computer code for the Jacobian matrix").
The Nasir/Zhou combination may not explicitly teach rescaling the given field development plan to scale of the target reservoir.
However, Roth teaches:
rescaling the given field development plan to scale of the target reservoir (Roth, [0254]: "In step (23), the 'merged data' data table is provided to a prediction algorithm that predicts well production for a variety of production intervals (e.g. 30-day, 60-day, 90-day) and production types (e.g. oil, gas, water, condensate) at different points in the well's lifecycle ( e.g. permitted, drilled, completed, flowback tested, producing, recompleted)" where Roth's field plans had been normalized previously to 30-day increments, as in [0172]: "production between wells cannot be properly compared without first normalizing the production, which in tum requires a reliable measurement of the Days On. In embodiments of the present disclosure, the production is normalized and interpolated so that 30-day increments of producing days, or Days On, can be compared between wells").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou combination regarding generating a field development plan for the target reservoir on the reservoir template with the rescaled and normalized target input values, the trained policy network, and the reservoir simulator with those of Roth regarding rescaling the generated field plan to scale of the target reservoir model to generate a final field plan for the target reservoir.
The motivation to do so would be to ensure that actual and forecasted production plans provide consistent information (Roth, [0196]: "FIG. 24 is a production data table showing normalized production attributes obtained both with and without use of the 'days on' estimation technique.... As evident from the table, the lower volumes in the non-corrected data of columns 744 and 748 could lead to wells being falsely considered as poor producers, when in fact they are not").
The Nasir/Zhou/Roth combination may not explicitly teach outputting, on a graphical user interface, at least a portion of the final field development plan.
However, Castellini teaches:
outputting, on a graphical user interface, at least a portion of the final field development plan (Castellini, [0077]: "FIG. 12 illustrates system 200 that can be used to perform the iterative response Surface methods. Such as computer-implemented method 100, previously described herein. System 200 includes user interface 210, such that an operator can actively input information and review operations of system 200. User interface 210 can be any means in which a person is capable of interacting with system 200 Such as a keyboard, mouse, touch-screen display, or Voice-command controls").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth combination regarding generating a final field development plan with the teachings of Castellini regarding outputting at least a portion of the final field development plan on a graphical user interface.
The motivation to do so would be to facilitate performance optimization or decision making by users (Castellini, [0084]: "A visual display can be produced, such as through reporting unit 250 or user interface 210. ... The displayed information can be utilized to forecast or optimize the production performance of the sub terranean reservoir, or used in making other reservoir management decisions").
Regarding Claim 26, Nasir teaches:
A system of generating a field development plan for a hydrocarbon field development (Nasir, p. 1, 1. Introduction: "The goal of this work is to investigate the performance of convolutional neural network-based (CNN) deep reinforcement learning (DRL) algorithm to the field development optimization problem (with fixed well operational settings)" and "we have the field development optimization (FDO) problem where decisions on the number of wells, their type (extraction/injection), location, drilling sequence, and well operational settings need to be made to optimally extract hydrocarbon in order to maximize an economic metric," where Nasir's optimal field development solution corresponds to the instant field development plan), the system comprising: one or more physical processors configured by machine-readable instructions (Nasir, p. 3, 4.2. DRL algorithm and CNN architecture: "A single Nvidia V100 GPU is used to train the CNN network. The RLlib package [10] handles the communication between the GPU and CPUs and the parallel execution of the flow simulation with the CPUs using a custom field development gym environment developed in this work," where Nasir's use of the RLib package corresponds to the instant instructions) to execute precisely those steps recited in the rejection of Claim 21.
Regarding Claim 24, the rejection of Claim 21 is incorporated.
The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising comparing the final field development plan for the target reservoir against at least one other field development plan for the target reservoir (Nasir, p. 5, 5. Results: "PSO-MADS algorithm is also used to benchmark the performance of PPO. ... As noted earlier, using PSO-MADS we obtain a single solution (and not a policy) that (potentially) maximizes the NPV" and p. 6, 5. Results: "Although PSO-MADS obtained a higher cumulative reward than that obtained by following the policy of the large network, the PSO-MADS solution includes a total of five wells while that of the large networks solution has four wells (less capital cost) and comparable NPV. This results suggests that the policy obtained by DRL are applicable for field development optimization," where Nasir's solutions correspond to the instant final plans), wherein the at least one other field development plan is generated by a human, by an optimization algorithm, or any combination thereof (Nasir, p. 1, Abstract: "The proximal policy optimization (PPO) algorithm is considered with two CNN architectures of varying number of layers and composition. Both networks obtained policies that provide satisfactory results when compared to a hybrid particle swarm optimization - mesh adaptive direct search (PSO-MADS) algorithm that has been shown to be effective at solving the FDO problem," where Nasir's PSO-MADS corresponds to the instant optimization algorithm).
Regarding Claim 25, the rejection of Claim 21 is incorporated.
The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the given field development plan comprises well counts, well locations, well type, well sequence, or any combination thereof (Nasir, p. 2, 3. Optimization Problem and Data: "Our goal in this work is to use DRL to find a policy (
π
) that maps from states (representation of the environment/reservoir model) to optimal actions/decisions for the field development problem" where p. 1, 1. Introduction: "In the field development problem, the production life-cycle is divided into a number of discrete drilling stages. Our goal in each stage is to decide if to drill a well or not. If the optimal decision is to drill a well, the decision on the well type and well location will also need to be made") to improve profitability of a hydrocarbon field development (Nasir, p. 2, 3. Optimization Problem and Data: "In this work, the cumulative reward
J
is defined as the net present value (NPV)" where p. 1, Abstract: "The field development optimization (FDO) problem represents a ... problem in which we seek to obtain the number of wells, their type, location, and drilling sequence that maximizes an economic metric").
Claims 6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of Schulman, et al., "Proximal Policy Optimization Algorithms" (hereinafter "Schulman").
Regarding Claim 6, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the deep reinforcement learning comprises proximal policy optimization (PPO) having a ... combination of four components (Nasir, p. 4, 4.2. DRL algorithm and CNN architecture: "The PPO with clipped surrogate objective (PPO-clip) is used in this work. The policy update for PPO-clip using multiple steps of stochastic gradient descent is given by:
PNG
media_image2.png
59
456
media_image2.png
Greyscale
where
L
is given by
PNG
media_image3.png
116
455
media_image3.png
Greyscale
," where Nasir's parameters of
L
s
,
a
,
θ
k
,
θ
correspond to the instant four components).
Nasir may not explicitly teach wherein the four components are (A) a policy loss
L
π
, (B) KL divergence penalty
L
k
l
, (C) a value function loss
L
v
f
, and (D) an entropy penalty
L
e
n
t
, and wherein the four components are expressed in an equation:
L
P
P
O
=
L
π
+
c
k
l
L
k
l
+
c
v
f
L
v
f
+
c
e
n
t
L
e
n
t
wherein
c
k
l
,
c
v
f
, and
c
e
n
t
are weights for each individual loss component.
However, Schulman teaches:
wherein the four components are (A) a policy loss
L
π
(Shulman, p. 3, 3 Clipped Surrogate Objective: the main objective of Eq. 7,
L
C
L
I
P
θ
), (B) KL divergence penalty
L
k
l
(Shulman, p. 4, 4 Adaptive KL Penalty Coefficient: Eq. 8, term
β
K
L
…
), (C) a value function loss
L
v
f
(Shulman, p. 5, 5 Algorithm: Eq. 9, term
c
1
L
t
V
F
θ
), and (D) an entropy penalty
L
e
n
t
(Shulman, p. 5, 5 Algorithm: Eq. 9, term
c
2
S
π
θ
s
t
), and wherein the four components are expressed in an equation:
L
P
P
O
=
L
π
+
c
k
l
L
k
l
+
c
v
f
L
v
f
+
c
e
n
t
L
e
n
t
wherein
c
k
l
,
c
v
f
, and
c
e
n
t
are weights for each individual loss component (Shulman, p. 5, 5 Algorithm: Eq. 9, where
β
,
c
1
, and
c
2
correspond to the instant weights).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding the deep reinforcement learning comprises proximal policy optimization (PPO) having a combination of four components with those of Shulman regarding the four components being a policy loss, a KL divergence penalty, a value function loss, and an entropy penalty, wherein the four components are expressed as a sum with weights for each individual loss component.
The motivation to do so would be to constrain model policy updates (Shulman, p. 3, 3 Clipped Surrogate Objective: "Without a constraint, maximization ... would lead to an excessively large policy update; hence, we now consider how to modify the objective, to penalize changes to the policy that move
r
t
θ
away from 1"), follow accepted modeling techniques (Shulman, p. 4, 5 Algorithm: "Most techniques for computing variance-reduced advantage-function estimators make use a learned state-value function
V
s
"), ensure sufficient exploration of the action space during learning (Shulman, p. 5, 5 Algorithm: "This objective can further be augmented by adding an entropy bonus to ensure sufficient exploration"), and provide basis for comparison against alternatives (Shulman, p. 4, 4 Adaptive KL Penalty Coefficient: "we found that the KL penalty performed worse than the clipped surrogate objective, however, we've included it here because it’s an important baseline").
Regarding Claim 9, the rejection of Claim 1 is incorporated.
The Nasir/Zhou/Roth/Castellini combination may not explicitly teach wherein the policy neural network and the value neural network do not share weights.
However, Shulman teaches:
wherein the policy neural network and the value neural network do not share weights (Shulman, p. 6, 6.1 Comparison of Surrogate Objectives: "To represent the policy, we used a fully-connected MLP with two hidden layers of 64 units, and tanh
nonlinearities, outputting the mean of a Gaussian distribution, with variable standard deviations, following [Sch+15b; Dua+16]. We don’t share parameters between the policy and value function").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding constructing a policy neural network and a value neural network that project a state with those of Shulman regarding the policy neural network and the value neural network not sharing weights.
The motivation to do so would be to facilitate development of models exhibiting simplicity and suitability to specialized domains (Shulman, p. 8, 7 Conclusion: "We have introduced proximal policy optimization, a family of policy optimization methods that use multiple epochs of stochastic gradient ascent to perform each policy update. These methods have the stability and reliability of trust-region methods but are much simpler to implement, requiring only few lines of code change to a vanilla policy gradient implementation, applicable in more general settings (for example, when using a joint architecture for the policy and value function)").
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of He, et al., "Deep Reinforcement Learning with a Natural Language Action Space" (hereinafter "He").
Regarding Claim 10, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the policy neural network and the value neural network comprise an action ... layer ... to learn ... actions during the training (Nasir, Fig. 2, and p. 4, 4.2. DRL algorithm and CNN architecture: "The individual concatenated tensors are further processed by fully connected layers to produce the probability distribution of the action space condition on the state space (
π
a
s
for the policy network)," where the layer "FC (# of actions)" as depicted by Fig. 2 is shared by the joint CNN architecture).
Nasir may not explicitly teach wherein the policy neural network and the value neural network comprise an action embedding layer to force the policy network to learn low dimensional representations of actions during the training.
However, He teaches:
wherein the policy neural network (He, p. 3, 2.4 Learning the DRRN: Back propagation: "To learn the DRRN [deep reinforcement relevance network], we use the 'experience-replay' strategy ..., which uses a fixed exploration policy to interact with the environment to obtain a sample trajectory," where He's DRRN corresponds to the instant policy neural network) ... comprise an action embedding layer (He, p. 3, 2.3 DRRN architecture: Forward activation: "Given any state/action text pair
s
t
,
a
t
i
, the DRRN estimates the Q-function
Q
s
t
,
a
t
i
in two steps. First, map both
s
t
and
a
t
i
to their embedding vectors using the corresponding DNNs, respectively. Second, approximate
Q
s
t
,
a
t
i
using an interaction function such as the inner product of the embedding vectors," where He's action embedding layer is depicted in Fig. 1(c) as
h
1
,
a
i
and
h
2
,
a
i
, but has a one-layer variant, as in p. 6, 3.2 Experiment setup: "We apply DRRNs with both 1 and 2 hidden layer structures") to force the policy network to learn low dimensional representations of actions during the training (He, p. 6, 3.2 Experiment setup: "We use DRRNs with 20, 50 and 100-dimension hidden layer(s) and build learning curves during experience-replay training" where the learnt action space is much greater than 20, as in: p. 2, 1 Introduction: "For actions described by natural language text strings, the action space is inherently discrete and potentially unbounded due to the exponential complexity of language with respect to sentence length").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding training the policy neural network and the value neural network using deep reinforcement learning with those of He regarding the policy neural network comprising an action embedding layer to force the policy network to learn low dimensional representations of actions during the training.
The motivation to do so would be to facilitate improved learning of complex action spaces and improved training efficiency (He, p. 9, 4 Related Work: "The overall action space is defined by the action-argument product space. This pre-specified product space is not feasible for the more complex text strings.... Our proposed DRRN, on the other hand, can handle the more complex text strings" and p. 9, 5 Conclusion: "We show that the DRRN converges faster and to a better solution for Q-learning than alternative architectures that do not use separate embeddings for the state and action spaces").
Claims 11 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of Tang, et al., "Implementing action mask in proximal policy optimization (PPO) algorithm" (hereinafter "Tang").
Regarding Claim 11, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising ... at least one user-defined ... action (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action)").
Nasir may not explicitly teach applying action masking to invalidate at least one ... invalid action during the training.
However, Tang teaches:
applying action masking to invalidate at least one ... invalid action during the training (Tang, p. 202, 2.2. Adding action mask: "To remove invalid actions from consideration, we use an action mask, which indicates each action as valid or invalid in each state. ... The environment needs to provide the mask to indicate which actions are invalid, and the agent then ignores those actions. To incorporate this change, the PPO algorithm in Fig. 1 needs ... modifications").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding training the policy neural network and the value neural network using deep reinforcement learning on the plurality of training reservoir models with those of Tang regarding applying action masking to invalidate at least one invalid action during training.
The motivation to do so would be to reduce training time (Tang, p. 201, 1. Introduction: "Our approach includes an action mask that indicates valid/invalid actions in each state, and a procedure to re-normalize the probability of valid actions. The results show that removing invalid actions from the action list in the PPO algorithm can indeed reduce the training time (or increase the return for the same number of training epoch)").
Regarding Claim 23, the rejection of Claim 21 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising ... at least one user-defined ... action during generating the field development plan for the target reservoir (Nasir, p. 6, Figure 5: "Optimal solutions (with an NPV of $269.06 million) obtained by following the policy of the large network after user defined initial action)").
The Nasir/Zhou/Roth/Castellini combination may not explicitly teach applying action masking to invalidate at least one ... invalid action during the training.
However, Tang teaches:
applying action masking to invalidate at least one ... invalid action (Tang, p. 202, 2.2. Adding action mask: "To remove invalid actions from consideration, we use an action mask, which indicates each action as valid or invalid in each state. ... The environment needs to provide the mask to indicate which actions are invalid, and the agent then ignores those actions. To incorporate this change, the PPO algorithm in Fig. 1 needs ... modifications").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Roth/Castellini combination regarding generating a field development plan for the target reservoir on the reservoir template with those of Tang regarding applying action masking to invalidate at least one invalid action. The motivation to do so would be to improve runtime performance of the trained model (Tang, p. 202, 3. Experiments and results: "We conduct two experiments to evaluate the performance of removing invalid actions. ... After training, plots of end-of-episode rewards for both versions are obtained from TensorBoard, as shown in Fig. 3. It is clearly seen that the agent in the proposed algorithm performs much better than its counterpart, especially when the number of training steps reaches 2 M steps").
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of Maucec, et al. (US 2021/0389491 A1, hereinafter "Maucec").
Regarding Claim 13, the rejection of Claim 1 is incorporated.
The Nasir/Zhou/Roth/Castellini combination may not explicitly teach wherein the policy neural network, the value neural network, or both comprise a graph neural network to represent a fault
However, Maucec teaches:
wherein the policy neural network, the value neural network, or both comprise a graph neural network to represent a fault (Maucec, [0016]: "embodiments of the disclosure include systems and methods for using a machine-learning algorithm to generate a graph neural network from a reservoir graph network. ... Thus, rather than simply using a grid model to perform simulations, a reservoir region in the grid model may also be represented as a graph network" and [0018]: "The hydrocarbon-bearing formation (104) may include a porous or fractured rock formation that resides underground, beneath the earth's surface" where [0075]: "Turning to FIG. 13, FIG. 13 illustrates a deep reinforcement learning (DRL) algorithm architecture in accordance with one or more embodiments. ... The DRL algorithm architecture (1300) may further include a state (s) that defines a concrete and immediate situation in which a particular agent finds itself (i.e., a prior version of a graph neural network may be the initial state, and the updated graph neural network may be a final state).... The DRL algorithm architecture (1300) may further include a policy (7r) that defines a predetermined strategy that the agent employs" and [0076]: "In some embodiments, the state vector of a DRL algorithm architecture (1300) corresponds to a reservoir graph network").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Nasir regarding constructing a policy neural network and a value neural network with those of Maucec regarding the policy neural network, the value neural network, or both comprising a graph neural network to represent a fault. The motivation to do so would be to facilitate modeling hydrocarbon reservoir by simulation with improved accuracy and/or efficiency (Maucec, [0017]: "in simulations of massive hydrocarbon reservoirs, simulation grid sizes routinely exceed hundreds of millions grid cells with numerous possible scenarios and realizations. Given this situation, a machine-learning model may provide a faster solution to simulating reservoirs. ... However, many predictive data models may be fast but inaccurate in contrast to full-physics models that may be accurate but slow. ... Thus, a graph neural network based on a geological region may provide a dynamic, interactive network of objects and relations for simulating and predicting data").
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of Nelson, et al. (U.S. 2017/0159402 A1, hereinafter "Nelson").
Regarding Claim 14, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
wherein the field development action comprises drilling ... as two consecutive actions, wherein the two consecutive actions comprise determining a location of a ... well (Nasir, p. 2, 2. Related Work: "The focus in this work is on the more challenging field development optimization problem in which the decision to be made include the number of wells, their locations, types and drilling sequence").
Nasir may not explicitly teach drilling a horizontal well as two consecutive actions, wherein the two consecutive actions comprise determining a location of a heel of the horizontal well and determining a location of a toe of the horizontal well.
However, Nelson teaches drilling a horizontal well as two consecutive actions, wherein the two consecutive actions comprise determining a location of a heel of the horizontal well and determining a location of a toe of the horizontal well (Nelson, [0031]: "Multiple stage fracturing, also known as multizone fracturing, proceeds by first dividing the areas to be stimulated into discrete intervals" and [0032]: "In a multi-zone fracturing operation, the first zone subjected to stimulation is the farthest from the ground or platform surface. ... In a horizontal wellbore, the first zone is closest to the toe while the second zone is closer to the heel," where Nelson's farthest and closest correspond to the instant determine locations).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Nasir regarding the field development action comprising drilling as two consecutive actions, wherein the two consecutive actions comprise determining a location of a well with those of Nelson regarding drilling a horizontal well the consecutive actions of determining a location of the heel and the toe. The motivation to do so would be to facilitate improving the efficiency of drilling horizontal wells (Nelson, Claim 1: "A method of enhancing the efficiency in the removal of debris from a wellbore penetrating a multi-zoned subterranean reservoir wherein the debris originates, at least in part, from a fluid-impermeable barrier separating perforated zones during a multi-zone fracturing operation").
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Nasir, "Deep Reinforcement Learning for Field Development Optimization" (hereinafter "Nasir") in view of Zhou, "Parallel general-purpose reservoir simulation with coupled reservoir models and multi-segment wells" (hereinafter "Zhou") in view of Roth, et al. (U.S. 2019/0361146 A1, hereinafter "Roth") in view of Castellini, et al., (U.S. 2012/0059641 A1, hereinafter "Castellini") in view of Espeholt, et al., "IMPALA: Scalable Distributed Deep-RL with Importance-Weighted Actor-Learner Architectures" (hereinafter "Espeholt").
Regarding Claim 16, the rejection of Claim 1 is incorporated. The Nasir/Zhou/Roth/Castellini combination teaches:
further comprising applying transfer ... to speed up the training of the policy neural network and the value neural network (Nasir, p. 2, 3. Optimization Problem and Data: "In this work ... AD-GPRS ... is used for the flow simulation.... A restart strategy is used to improve the efficiency of the overall flow simulation. This entails saving information of previous flow simulations (due to actions taken in previous drilling stages) that allows for the simulation to be restarted at the beginning of the next drilling stage without the need to run the flow simulation from scratch").
Nasir may not explicitly teach applying transfer reinforcement learning to speed up the training of the policy neural network and the value neural network.
However, Espeholt teaches applying transfer reinforcement learning to speed up the training of the policy neural network and the value neural network (Espeholt, p. 8, 6. Conclusion: "our experiments on DMLab-30 show that, in the multi-task setting, positive transfer between individual tasks lead IMPALA to achieve better performance compared to the expert training setting" and p. 1, Abstract: "We demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30," where the policy and value functions of, for example, p. 4, 4.2. Actor-Critic algorithm: "Consider a parametric representation
V
θ
of the value function and the current policy
π
ω
. ... At training time
s
, the value parameters
θ
are updated by gradient descent ... and the policy parameters
ω
in the direction of the policy gradient" are depicted as neural networks in Fig. 3).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Nasir/Zhou/Roth/Castellini combination regarding applying transfer to speed up the training of the policy neural network and the value neural network with those of Espeholt regarding applying transfer reinforcement learning to speed up the training of the policy neural network and the value neural network.
The motivation to do so would be to enable more efficient use of computing resources and facilitate flexibility in neural network modeling (Espeholt, p. 1, 1. Introduction: "IMPALA achieves exceptionally high data throughput rates .... Crucially, IMPALA is also more data efficient than A3C based agents and more robust to hyperparameter values and network architectures, allowing it to make better use of deeper neural networks").
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122