Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
1. This action is in response to application amendments filed on 12-10-2025.
2. Claims 1 - 6, 8 - 12, 15 - 23 are pending. Claims 1, 11 have been amended. Claim 23 is new. Claims 7, 13, 14 have been canceled. Claims 1, 11 are independent. This application was filed on 5-10-2021.
Response to Arguments
3. Applicant’s arguments, see Arguments/Remarks Made in an Amendment, filed 12-10-2025, with respect to the rejection(s) under Schwartz in view of Comeaux have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Schwartz in view of Comeaux and further in view of Chong.
A. Applicant argues on page 1 of Remarks: ... when a new alert is detected, updates the learning model with a predefined negative reward; if no alert is detected in response to the target action, updates the learning model with a predefined positive reward ... .
The Examiner respectfully disagrees. Chong discloses positive rewards occurring and negative rewards occurring. Chong discloses when a “good” corrective action occurs such as no alert a positive reward is implemented, when a “bad” corrective action occurs such as an alert, a negative reward is implemented. (see Chong paragraph [0195]: reward value associated with a generated corrective action may determine whether the model/system is heading in the right direction, where a positive reward indicates a “good” corrective action (no new alert) and a negative reward indicates a “bad” corrective action (a new alert); (an alert is a bad corrective action (negative reward), and no alert is a good corrective action (positive reward))
B. Applicant argues on page 2 of Remarks: ... Schwartz does not contemplate, disclose, or suggest a predefined negative reward in response to a detected alert, and a predefined positive reward otherwise. The Comeaux and Chow references also do not disclose or suggest predefined rewards as set forth in the claims.
The Examiner respectfully disagrees. Chong discloses positive rewards occurring and negative rewards occurring. Chong discloses when a “good” corrective action occurs such as no alert a positive reward is implemented, when a “bad” corrective action occurs such as an alert, a negative reward is implemented. (see Chong paragraph [0195]: reward value associated with a generated corrective action may determine whether the model/system is heading in the right direction, where a positive reward indicates a “good” corrective action (no new alert) and a negative reward indicates a “bad” corrective action (a new alert); (an alert is a bad corrective action (negative reward), and no alert is a good corrective action (positive reward))
C. Applicant argues on page 2 of Remarks: ... For at least these reasons, Applicant submits that independent claims 1 and 11 are patentable under § 103.
Independent claim 11 has similar limitations as independent claim 1. Responses to arguments against independent claim 1 also answer arguments against independent claim 11.
D. Applicant argues on page 2 of Remarks: ... The remaining dependent claims are patentable at least because they depend from patentable base claims.
Responses to arguments against the independent claims also answer arguments against the associated dependent claims.
E. Applicant argues on page 2 of Remarks: New Claim 23.
Claim 23 is disclosed in the current Office Action.
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1 - 6, 10 - 12, 15 - 17, 20 - 22 are rejected under 35 U.S.C. 103 as being unpatentable over Schwartz et al. (Title: Autonomous Penetration Testing using Reinforcement Learning, dated: 16th November, 2018, School of Information Technology and Electrical Engineering University of Queensland) in view of Comeaux et al. (US Patent No. 10,878,428) and further in view of Chong et al. (Patent No. WO 2020172447).
Regarding Claims 1, 11, Schwartz discloses a method for identifying vulnerabilities in a security information and event management (SIEM) system and a system, comprising:
a) initializing a security testing agent (STA) with a test scenario goal and a reinforcement learning model, wherein the model defines a set of states indicative of progress towards the goal, a set of actions that can be taken by a legitimate user within a target environment, and a set of reward values associated with taking a specified action in a specified state; (see Schwartz page 15, lines 20-23: evaluating the security of a computer system utilizing penetration testing (pentesting); Pentesting involves performing an authorized controlled attack on a system in order to find any security vulnerabilities that could be exploited by an attacker; page 17, lines 8-11: Reinforcement Learning (RL): RL requires only the state space representation, the set of actions that can be performed and a reward function which defines what the RL agent is trying to achieve; The agent then learns a policy of actions to take from any given state through interaction with its environment) and
b) learning a policy to achieve the goal within the target environment (see Schwartz page 17, lines 9-10: The agent then learns a policy of actions to take from any given state through interaction with its environment),
wherein the learning includes a process that:
c) selects and takes a target action from the set of actions for a current state; (see Schwartz page 21, lines 20-23: At each time step the system will be in some state, s € S, and the agent will perform some action, resulting in two things: (1) a transition to a new state s’, where the new state is determined by the transition function, (2) a reward as determined by the reward function)
d) monitors for an alert triggered in response to the target action being taken within the target environment; (see Schwartz page 19, lines 12-15: utilizing tools such as traffic monitoring, port scanning and operating system (OS) detection in order to collect relevant information that can be used to determine if the system contains a vulnerability that can be exploited; page 21, lines 20-23: At each time step the system will be in some state, s € S, and the agent will perform some action)
h) calculates and saves an updated reward value in the model; (see Schwartz page 33, lines 23-24: The reward is defined over a transition R (s, a, s’), so starting from one state s, taking action a, and ending in the resulting state s’) and
i) in response to the process not being terminated, repeats the process for a next state. (see Schwartz page 36, lines 6-12: It takes an action and performs the transition given the current state of the simulator and returns the next state, reward for performing the action and whether the goal has been reached ... cycle of training: i) resetting the NAS to get the start state, ii) using the state to choose an action, iii) executing the action against the environment using the step function to receive the next state and reward, iv) repeat steps (ii) and (iii) until the goal is reached)
Schwartz does not specifically disclose for e) stores information about alert in an alert data store if an alert is detected, and for f) once target action has been undertaken, polls alert data store for information relating to new alerts.
However, Comeaux discloses:
e) stores information about the alert in an alert data store if an alert is detected; (see Comeaux col 4, lines 27-38: The alerts may be stored into a system database to wait further processing) and
f) once the target action has been undertaken, polls the alert data store for information relating to new alerts. (see Comeaux col 4, lines 28-28: Alert-generating systems and apparatuses described herein generate alerts based on inputs from one or more servers that are operable to track, monitor, and flag for potentially fraudulent and malicious network activity; The alerts contain various data fields indicating threats of fraud or attempts to penetrate an enterprise network, and indicating one or more particular users of the system associated with an alert (often the target of a type of fraud); The alerts may be stored into a system database to wait further processing; col 4, lines 5-8: Analyst computers may query and fetch these alerts from a database and may present the alerts to be addressed by an analyst, for example, according to a priority level of the respective alerts)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartz for e) stores information about alert in an alert data store if an alert is detected, and for f) once target action has been undertaken, polls alert data store for information relating to new alerts as taught by Comeaux. One of ordinary skill in the art would have been motivated to employ the teachings of Comeaux for the benefits achieved from the enhanced security solutions by identifying malicious network activity and then generating alerts to address the identified. (see Comeaux col 2, lines 5-8)
Schwartz-Comeaux does not specifically disclose for g) when a new alert is detected, updates the learning model with a predefined negative reward, and if no alert is detected in response to the target action, updates the learning model with a predefined positive reward.
However, Chong discloses:
g) when a new alert is detected, updates the learning model with a predefined negative reward; if no alert is detected in response to the target action, updates the learning model with a predefined positive reward. (see Chong paragraph [0195]: reward value associated with a generated corrective action may determine whether the model/system is heading in the right direction, where a positive reward indicates a “good” corrective action (no new alert) and a negative reward indicates a “bad” corrective action. (a new alert); (an alert is a bad corrective action (negative reward), and no alert is a good corrective action (positive reward))
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartz for g) when a new alert is detected, updates the learning model with a predefined negative reward, and if no alert is detected in response to the target action, updates the learning model with a predefined positive reward as taught by Chong. One of ordinary skill in the art would have been motivated to employ the teachings of Chong for the benefits achieved from the flexibility of a system that enables the presence of absence of alerts is associated with positive and negative rewards (feedback). (see Chong paragraph [0195])
Furthermore, for Claim 11, Schwartz discloses wherein a memory; and a processor coupled to the memory and configured to identify vulnerabilities associated with user actions in a security information and event management (SIEM) system, wherein identifying vulnerabilities includes performing operations. (see Schwartz page 61, lines 17-18: All experiments were conducted on single core on a personal laptop computer; (personal computer indicates a processor coupled to a memory for instruction retrieval and instruction execution))
Regarding Claims 2, 12, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein, in response to the goal being achieved, outputting a saved model and policy, wherein the policy includes the actions taken to achieve the goal. (see Schwartz page 49, lines 2-6: RL algorithms learn optimal policies through interaction with the environment; starting from some initial state/action, policy then iteratively learning the values of taking a certain action for a given state, Q(s, a), by choosing an action based on the current policy, applying that action to the environment, then updating the state-action value (save state-action parameters), (Q(s, a), based on the received experience (reward))
Regarding Claims 3, Schwartz-Comeaux-Chong discloses the method of claim 2, further comprising learning a new policy using the saved model. (see Schwartz page 17, lines 9-11 The agent then learns a policy of actions to take from any given state through interaction with its environment)
Regarding Claims 4, Schwartz-Comeaux-Chong discloses the method of claim 1, wherein the process restarts in response to detection of the alert. (see Schwartz page 19, lines 12-15: utilizing tools such as traffic monitoring, port scanning and operating system (OS) detection in order to collect relevant information that can be used to determine if the system contains a vulnerability that can be exploited; page 17, lines 8-11: Reinforcement Learning (RL): RL requires only the state space representation, the set of actions that can be performed and a reward function which defines what the RL agent is trying to achieve;)
Regarding Claims 5, 15, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein the actions are selected from a group consisting of: deleting files, downloading content, accessing content, data exfiltration, unauthorized use of an information technology (IT) service, unauthorized use of data, and malicious acts targeting availability or reliability of an IT service. (see Schwartz page 30, lines 1-9: Firewalls act to control which services can be communicated with on machines in a given subnet from any other connection point outside of the subnet; They function to allow certain services to be used and accessed from machines within a subnet with the correct permissions, while blocking access to that service from unwanted entry points; Each firewall is defined by a set of rules which dictate which service traffic is permitted for each direction along a connection between any two subnets or from the external network; (selected: accessing content, utilizing network communications))
Regarding Claims 6, 16, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein the target action is selected by either making a random selection or selecting the action having a highest reward value. (see Schwartz page 49, lines 2-6: RL algorithms learn optimal policies through interaction with the environment; This done by starting from some initial, typically random, policy then iteratively learning the values of taking a certain action for a given state, Q(s, a), by choosing an action based on the current policy, applying that action to the environment, then updating the state-action value, (Q(s, a), based on the received experience; page 47, lines 15-17: value function specifies the value of a state over the long run, such that the value of a state, s, (i.e. V(s)) is the total accumulated reward the agent can expect to receive in the long term starting from that state; (optimal policy indicates action with highest reward))
Regarding Claims 10, 20, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein initialization further includes defining a set of procedures for accessing an endpoint within the target environment to take actions. (see Schwartz page 47, lines 11-13: The policy is a mapping from the state space, S, to the action space, A; the agent to find the optimal policy n* that chooses the action, a, from any state, s, that maximizes the total expected discounted reward; reward defines the immediate reward for the current state and is sent by the environment on each time step)
Regarding Claim 17, Schwartz-Comeaux-Chong discloses the system of claim 16, wherein the target action is further selected based on a feasibility of taking an available action within the target environment. (see Schwartz page 33, lines 2-3: The action space, A, is the set of available actions within the NAS and includes a single scan action and an exploit for each service and each machine on the network)
Regarding Claims 21, 22, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein initializing the STA further comprises: wherein the metadata defines how to undertake actions. (see Schwartz page 15, lines 20-23: evaluating the security of a computer system utilizing penetration testing (pentesting); Pentesting involves performing an authorized controlled attack on a system in order to find any security vulnerabilities that could be exploited by an attacker; page 17, lines 8-11: Reinforcement Learning (RL): RL requires only the state space representation, the set of actions that can be performed and a reward function which defines what the RL agent is trying to achieve; The agent then learns a policy of actions to take from any given state through interaction with its environment)
Schwartz does not specifically disclose accessing an STA database, and retrieving metadata from database.
However, Comeaux discloses wherein accessing an STA database; and retrieving metadata from the STA database. (see Comeaux col 4, lines 28-28: Alert-generating systems and apparatuses described herein generate alerts based on inputs from one or more servers that are operable to track, monitor, and flag for potentially fraudulent and malicious network activity; The alerts contain various data fields indicating threats of fraud or attempts to penetrate an enterprise network, and indicating one or more particular users of the system associated with an alert (often the target of a type of fraud); The alerts may be stored into a system database to wait further processing; col 4, lines 5-8: Analyst computers may query and fetch these alerts from a database and may present the alerts to be addressed by an analyst, for example, according to a priority level of the respective alerts)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartz for accessing an STA database, and retrieving metadata from database as taught by Comeaux. One of ordinary skill in the art would have been motivated to employ the teachings of Comeaux for the benefits achieved from the enhanced security solutions by identifying malicious network activity and then generating alerts to address the identified. (see Comeaux col 2, lines 5-8)
6. Claims 8, 9, 18, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Schwartz in view of Comeaux and further in view of Chong and Fairbank et al. (US Patent No. 10,417,556).
Regarding Claims 8, 18, Schwartz-Comeaux-Chong discloses the method of claim 1 and the system of claim 11, wherein the reinforcement learning model includes q-learning model. (see Schwartz page 48, lines 13-20: using Q-learning which is a RL algorithm for learning an optimal policy; It relies on using experience to learn the Q-value function, Q(s, a),which tells the agent the expected reward if they perform action a from state s)
Schwartz does not explicitly disclose model implemented as a Q matrix with the set of states defined along a first axis (row: state) and the set of actions defined along a second axis (column: action), wherein each entry in the Q matrix includes an associated reward value.
However, Fairbank discloses wherein model implemented as a Q matrix with the set of states defined along a first axis and the set of actions defined along a second axis, wherein each entry in the Q matrix includes an associated reward value. (see Fairbank col 26, lines 16-25: a Q-table can be generated by the simulation that can include reward values; a Q-table can be thought of as a matrix, where each row of the matrix represents a different unique state, and each column represents a given action that can be taken, given that state; The value contained in the matrix for a given action and given state is known as the Q-value, and represents the relative expected reward if a given action is taken during a given state)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartz for model implemented as a Q matrix with the set of states defined along a first axis and the set of actions defined along a second axis, wherein each entry in the Q matrix includes an associated reward value as taught by Fairbank. One of ordinary skill in the art would have been motivated to employ the teachings of Fairbank for the benefits achieved from enabling a system that utilizes a table data structure with one axis associated with a state and a second axis associated with actions and intersection associated with a reward. (see Fairbank col 26, lines 16-25)
Regarding Claims 9, 19, Schwartz-Comeaux-Chong discloses the method of claim 7 and the system of claim 17, wherein a value of the predetermined negative reward and/or predetermined positive reward is updated in a matrix according to the formula:
Qnew(s,a) = Q(s,a) + α(R(s,a) +y max Q(s',a) - Q(s,a))
where α is a learning rate with O <a<=1,
R(s,a) is a reward received after completing the target action a at the current state s,
s’ is the next candidate state, and
y is a discount factor with O <=y <=1. (see Schwartz page 21, lines 20-23: At each time step the system will be in some state, s € &, and the agent will perform some action, resulting in two things: (1) a transition to a new state s’, where the new state is determined by the transition function, (2) a reward as determined by the reward function; page 21, lines 20-23: At each time step the system will be in some state, s € &, and the agent will perform some action, resulting in two things: (1) a transition to a new state s’, where the new state is determined by the transition function, (2) a reward as determined by the reward function; page 47, lines 15-17: value function specifies the value of a state over the long run, such that the value of a state, s, (i.e. V(s)) is the total accumulated reward the agent can expect to receive in the long term starting from that state)
Chong discloses a positive reward or a negative reward as stated above.
Fairbank discloses a Q Matrix data structure as stated above.
7. Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Schwartz in view of Comeaux and further in view of Chong and Bisson et al. (US Patent No. 8,078,653).
Regarding Claim 23, Schwartz-Comeaux-Chong discloses the method of claim 1.
Schwartz does not specifically disclose generating an index of file sizes to full file paths and constraining the set of actions to a set of feasible actions by determining if one of more actions in the set of actions relate to operations on files listed in the index.
However, Bisson discloses wherein further comprising: generating an inverted index of file sizes to full file paths; and constraining the set of actions to a set of feasible actions by determining if one of more actions in the set of actions relate to operations on files listed in the inverted index. (see Bisson cols 7-8: Metadata are data describing the entity of a hierarchical storage structure. Metadata about an entity can usually be obtained from the networked file system or the hierarchical storage structure, without physically accessing the content of the entity. Examples of the metadata that can be extracted from a file include: mode number; created flag, modified flag, deleted flag, full path name, file extension, file size, soft or hard link, file or directory, user id, group id; hard link count; create time, access time, or modified time, etc. Metadata for a block can include block id, size, location, access information, etc. Alternatively, an additional call can be made to open an entity and access the content of the entity. For example, specific operations can be performed to specific types of files, e.g., image, audio, video files, etc, to extract metadata embedded in these special files.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Jusko for generating an index of file sizes to full file paths and constraining the set of actions to a set of feasible actions by determining if one of more actions in the set of actions relate to operations on files listed in the index as taught by Bisson. One of ordinary skill in the art would have been motivated to employ the teachings of Bisson for the flexibility of a system that enables multiple parameters to be utilized for data processing such as file size and file path information. (see Bisson cols 7-8)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARLTON JOHNSON whose telephone number is (571)270-1032. The examiner can normally be reached Work: 12-9PM (most days).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shewaye Gelagay can be reached on 571-272-4219. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CJ/
March 23, 2026
/KHOI V LE/Primary Examiner, Art Unit 2436
Prosecution Insights