Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The response filed 8/6/2026 was received and considered.
Claims 1-20 are pending.
Response to Arguments
Applicants’ arguments with respect to claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s remarks (pp. 11-12, regarding rejections under 34 U.S.C. §112) are persuasive in view of the amendments. The rejections are withdrawn.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over CN 115270133 A to Li et al. (Li).
Regarding claim 1, Li discloses a method for generating a cyberattack sequence based on reinforcement learning, the method comprising: generating a cyberattack simulation environment including an action space (attacker assesses the current state of the test environment, the set of attack actions, and the attack targets, ¶115), the action space comprising a plurality of actions (set of attack actions, ¶115), each action being defined by a combination of a host (target host, ¶89) and an attack technique (attack actions, ¶89), wherein the attack technique includes: pre-attack state information defining a state of the host required for execution of the attack technique; and post-attack state information defining a state of the host to be updated upon successful execution of the attack technique (for each action, two essential properties are required: (1) A set of preconditions must be true for an action to be performed; (2) a set of postconditions that will be true after the action is performed, ¶¶92-94), wherein each of the pre-attack state information and the post-attack state information comprises one or more key-value pairs, each key identifying a type of a host-state feature and each corresponding value specifying a value of the host-state feature (for example, runExploit(X) means to exploit vulnerability E on host X, where a prerequisite is that host X is vulnerable to vulnerability E and a postcondition is that the attacker establishes a foothold on host X, ¶n0021); determining whether a current state of the host satisfies the pre-attack state information (preconditions must be true before the attacker's capability is executed, ¶107); in response to determining that the current state of the host satisfies the pre-attack state information, executing the attack technique (preconditions must be true before the attacker's capability is executed, ¶107) and, upon successful execution of the attack technique, updating the current state of the host based on the post-attack state information (the postconditions will be true after the attacker's capability is executed, ¶107); training a cyberattack agent model based on interactions with the cyberattack simulation environment (Equation (2), represents the scoring function of the attack plan, is an action score value assigned by the designer to each independent action, ¶127; uses the reinforcement learning Q-Learning algorithm to automatically construct red team attack plans, ¶n0029); and generating an attack sequence using the trained cyberattack agent model (invention provides a method for generating red team test plans, ¶n0029).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 6-13 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Li, as applied to claim 1, in view of US 2017/0006055 A1 to Strom et al. (Strom).
Regarding claim 11, the claim is similar in scope to claim 1 and is therefore rejected using a similar rationale, but lacks a memory storing at least one program and at least one processor configured to execute the at least one program to perform the method. However, Strom teaches that it was known to construct a server to perform a cyberattack simulation method, the server including processors, memory and one or more programs to perform the method (¶26). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify Li such that the method is performed by an apparatus including a memory storing at least one program and at least one processor configured to execute the at least one program to perform the method. One of ordinary skill in the art would have been motivated to perform such a modification to utilize known computing technology to implement the invention, as taught by Strom.
Regarding claims 2 and 12, Li discloses wherein the cyberattack simulation environment further includes a state space (Using a Markov decision model, at time step and state , an action is performed, which will transition to the next state, ¶n007), and a reward function (If state has a positive effect on the attack plan, the attacker will be rewarded; otherwise, the attacker will be punished, ¶n007), but lacks a network model. However, Strom, in an analogous art (simulating network attacks), teaches an attack simulation with preconditions and postconditions (¶138), including maintaining a network model (logic engine may generate and maintain a model of the target network or portions of the target network that the engine uses to determine which action or actions to select, ¶142). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify Li such that the cyberattack simulation environment further includes a network model. One of ordinary skill in the art would have been motivated to perform such a modification to determine which actions to select based on the current state of the network, as taught by Strom.
Regarding claims 3 and 13, Li, as modified, teaches wherein generating the cyberattack simulation environment comprises generating the cyberattack simulation environment by receiving a predefined simulation scenario configuration file; and the predefined simulation scenario configuration file includes network configuration information (as modified by Strom, ¶142), host asset configuration information (maintains intelligence information from target host, ¶35), and information defining the attack techniques (`tactic` field is the corresponding tactic of the adversary's attack technique in the ATT&CK attack behavior knowledge base and model, ¶n0006).
Regarding claims 6 and 16, Li, as modified, teaches wherein the state space includes: respective current states of a plurality of hosts constituting a network (information gathering module is used to collect attack surface and intelligence information from the target host, analyze and generate vulnerability information on the target host, and transmit the attack surface, intelligence, and vulnerability information to the attack plan decision module, ¶35); and an execution-result state indicating a result of execution of a selected attack technique (postconditions that will be true after the action is performed, ¶94).
Regarding claims 7 and 17, Li discloses wherein the reward function calculates a reward based on an allocation value of a host newly compromised as a result of a change in the state space (if state has a positive effect on the attack plan, the attacker will be rewarded; otherwise, the attacker will be punished, ¶n0017).
Regarding claims 8 and 18, Li discloses wherein training the cyberattack agent model comprises: generating, by the cyberattack agent model, a candidate cyberattack sequence through interaction with the cyberattack simulation environment (using a Markov decision model, at time step and state, executing action will transition to the next state, where is the precondition for action and is the postcondition for action, ¶n0017) and; updating the cyberattack agent model based on rewards obtained from state transitions caused by execution of attack techniques included in the candidate cyberattack sequence (if state has a positive effect on the attack plan, the attacker will be rewarded; otherwise, the attacker will be punished; in subsequent training, rewards and punishments will cause attackers in a specific state to choose actions that promote the attack plan and avoid actions that are meaningless to the attack plan ¶n0017).
Regarding claims 9 and 19, Li discloses wherein training the cyberattack agent model further comprises: analyzing the candidate cyberattack sequence; for an attack technique included in the candidate cyberattack sequence, comparing a current state of a target host with the pre-attack state information included in the attack technique (information gathering module is used to collect attack surface and intelligence information from the target host, analyze and generate vulnerability information on the target host, and transmit the attack surface, intelligence, and vulnerability information to the attack plan decision module, ¶35; preconditions must be true before the attacker's capability is executed, ¶107); and only when the current state of the target host satisfies the pre-attack state information and the attack technique is successfully executed (preconditions must be true before the attacker's capability is executed, ¶107), updating the current state of the target host based on the post-attack state information included in the attack technique (the postconditions will be true after the attacker's capability is executed, ¶107).
Regarding claims 10 and 20, Li discloses wherein the attack technique corresponds to an attack technique of a MITRE ATT&CK framework (each capability is designed based on adversary attack tactics and techniques in the ATT&CK attack behavior knowledge base and model, ¶15).
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Li and Strom, as applied to claims 2 and 12, in view of CN 115766113 A to Chen et al. (Chen) and “Applying reinforcement learning for enhanced cybersecurity against adversarial simulation” by Oh et al. (Oh).
Regarding claims 4 and 14, Li discloses wherein the network model is configured with a host (intelligence information from a target host, ¶35), but lacks wherein the network model includes a plurality of subnetworks, a network topology, a plurality of hosts, and at least one firewall, and an allocation value used in calculating a reward is assigned to each of at least one of the plurality of hosts. However, Chen, in an analogous art (network penetration testing, ¶58), teaches that it was known to utilize a target network environment model described using configurations of a host, specifically including subnets, network topology, firewalls, routers, etc. (¶58). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to further modify Li such that the network model includes a plurality of subnetworks, a network topology, a plurality of hosts, and at least one firewall. One of ordinary skill in the art would have been motivated to perform such a modification to describe the state of a given target host based on a position within the target network, as taught by Chen. As modified, Li lacks where an allocation value used in calculating a reward is assigned to each of at least one of the plurality of hosts. However, Oh, in an analogous art (reinforcement learning for cyberattacks), teaches that it was known to provide a reward for a successful target compromise, where the reward is based on the importance of the target host (p. 5, §2.3, ¶1). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to further modify Li such that an allocation value used in calculating a reward is assigned to each of at least one of the plurality of hosts. One of ordinary skill in the art would have been motivated to perform such a modification to utilize a reward based on the value of a target, as taught by Oh.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Li and Strom, as applied to claims 2 and 12, in view of “Autonomous Penetration Testing Based on Improved Deep Q-Network”, by Zhou et a. (Zhou).
Regarding claims 5 and 15, Li, as modified, lacks wherein the reward function is defined as:
R
s
t
+
1
,
a
t
,
s
t
=
v
a
l
u
e
s
t
+
1
,
s
t
-
c
o
s
t
(
a
t
)
, wherein
v
a
l
u
e
s
t
+
1
,
s
t
represents an allocation value of a host newly compromised during a transition from state
s
t
to state
s
t
+
1
resulting from execution of action
a
t
, and
c
o
s
t
(
a
t
)
represents an execution cost of action. However, Zhou, in an analogous art (penetration testing), teaches that it was known to conduct an exploit on a target (p. 4, §2.3) and define a reward function
R
=
∑
h
∈
H
v
a
l
u
e
h
-
∑
a
∈
A
c
o
s
t
(
a
)
(p. 4, equation (5)), where
h
represents the compromised hosts,
v
a
l
u
e
h
represents the importance of any hosts and
c
o
s
t
(
a
)
represents of the cost of action
a
(p. 4, §2.3), to enable calculation of a reward
R
that mimics hackers in the real world (p. 4, §2.3, ¶1). Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify Li such that the reward function is defined as:
R
s
t
+
1
,
a
t
,
s
t
=
v
a
l
u
e
s
t
+
1
,
s
t
-
c
o
s
t
(
a
t
)
, wherein
v
a
l
u
e
s
t
+
1
,
s
t
represents an allocation value of a host newly compromised during a transition from state
s
t
to state
s
t
+
1
resulting from execution of action
a
t
, and
c
o
s
t
(
a
t
)
represents an execution cost of action. One of ordinary skill in the art would have been motivated to perform such a modification to utilize a known method of calculating the reward function of applying an exploit, as taught by Zhou.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J SIMITOSKI whose telephone number is (571)272-3841. The examiner can normally be reached on Monday - Friday, 7:00-3:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carl Colin can be reached on 571-272-38623862. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Michael Simitoski/ Primary Examiner, Art Unit 2493
September 22, 2026
