DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-22 filed 3/24/2026.
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/24/2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 9-11, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation by reference of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016).
For claim 1, Simpson discloses: a network interface device (§3 ¶1-2, figs.1-3 contemplates implementation on SmartNIC devices, see also §4.1-2 giving hardware overview) comprising:
a memory to store configuration values (§4.1-2) associated with a reinforcement learning (RL) routine and a set of RL-related parameters of the RL routine (fig.1 shows RL routine and associated parameters, e.g., state, reward, configuration data);
packet processing circuitry to receive network packets (§4.1-2, fig.1 shows incoming packets received by the P4 pipeline); and
accelerator circuitry coupled to the memory, the packet processing circuitry (§4.1-2 gives accelerator processor circuit overview), and a host processing device (§4.2), wherein the accelerator circuitry is to:
detect a network packet (§2.3 ¶2, §5.1 ¶1 contemplates use of algorithms for detecting particular packet header state data in association with actions, such as with various flow preference levels, such as for DDoS defense; see also §4.2 ¶2-3 describing determining packet state in the context of implementing DDoS RL control application, the state including various tilings on a 20-dim state vector and 10 possible actions corresponding to limiting traffic flow via incorporation of Simpson’s earlier work “Per-Host DDoS Mitigation by Direct-Control Reinforcement Learning”, particularly §III.C “Feature Space”: Table 1 describing the feature space including IP address, §IV: “Rethinking the State Space” describing the rationale for incorporating various features such as IP address; hence, a packet and its corresponding state is detected) ; and
execute the RL routine, using the configuration values and in response to detecting the network packet (ibid: the RL routine is executed based on the detected packet and state), to update a data structure storing action values for discrete states (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) and to employ observation information derived from or associated with the network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment / preference levels, dropping packets, changing rate limits)
cause, on behalf of the host processing device, the RL routine to perform further reinforcement learning via updates to one or more of the configuration values and action values and responsive to inferences made based on the RL-related action (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism modes of fig.3a-b; hence, based on activity rewards (See Alg.1: lookup_reward_from_key), the configuration value tiles are updated; see §3.2 ¶1 disclosing the updating of action values, i.e., the determining of a current action and sending it out into the environment); and
continue to process additional network packets using one or more updated configuration values and action values, through the RL routine on behalf of the host processing device (fig.3a-b, Alg.1: rewards, policy, and updating continues using updated tiling configurations; §3.2 ¶1 disclosing updating and executing action values).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
For claim 2, Simpson modified by Viegas discloses the unit of claim 1, as described above. Simpson modified by Viegas further discloses: wherein the network packets are further processed through a second RL routine based on a particular criterion comprising containing a particular identifier comprising one of a destination address or a fixed byte portion of each network packet (§4.2 last ¶ discloses various extracted features including IP address (see “DDOS” Table 1, §IV); see also §2.3 ¶2 contemplating taking header packet data as input, such as for a preference or priority list policy, the packet header data containing packet identifiers such as source, destination, the data being a fixed byte portion of a packet, the data being used to track flow state such as described in §5.1 ¶1, with §5.2 ¶2, §5.1 ¶1-3 contemplating deployments across a network).
For claim 9, Simpson discloses: a data processing unit comprising:
a network interface card (NIC) (§3 ¶1-2, figs.1-3 contemplates implementation on SmartNIC devices, see also §4.1 giving hardware overview) comprising:
a memory to store configuration values (§4.1) associated with a reinforcement learning (RL) routine and a set of RL-related parameters for implementing the RL routine (fig.1 shows RL routine and associated parameters, e.g., state, reward, configuration data);
a network interface to receive network packets (§4.1, fig.1 shows incoming packets received by the P4 pipeline); and
an accelerator circuit coupled to the memory and the network interface (§4.1-4.2 gives accelerator processor circuit overview), the accelerator circuit to iteratively:
detect a network packet that is received (§5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense); and
execute the RL routine, using the configuration values and in response to detecting the network packet (ibid: the RL routine is executed), to update a data structure storing action values for discrete states (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) to employ observation information derived from or associated with a network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment levels, dropping packets, changing rate limits; §5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense, with §4.2 describing the packet in the context of total state including various tiling on a 20-dim state vector and 10 possible actions corresponding to limiting traffic flow via incorporation of Simpson’s earlier work “Per-Host DDoS Mitigation by Direct-Control Reinforcement Learning”, particularly §III.C “Feature Space”: Table 1 describing the feature space, e.g., including IP address, §IV: “Rethinking the State Space” describing the rationale for incorporating various features such as IP address; hence, a packet and its corresponding state is detected and observed to perform RL actions); and
a host processing device coupled with the NIC and the accelerator circuit (§3.5 contemplates reconfiguration via a host computer via control packets during runtime), the host processing device to:
send an interrupt request to an RL configuration interface of the accelerator circuit to cause a context switch of the accelerator circuit to an application associated with the RL routine (ibid: sending learning parameters, policy design changes such as for a new policy for another problem, hence, interrupting a currently executing policy for another problem, hence, causing a context change; these learning parameters being received by the configuration interface of the NIC, such as via UDP and reserved values (§3.5 ¶1)); and
send a configuration file to the NIC, the configuration file to cause the accelerator circuit to program the configuration values, the action values, and associated formatting data into the memory (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, action values, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose: wherein the packet is received at a particular port of a plurality of a plurality of ports of the NIC.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
For claim 10, Simpson modified by Viegas discloses the unit of claim 9, as described above. Simpson further discloses: wherein the memory comprises:
one or more software registers to store the configuration values (Simpson fig.1b, §3.4 ¶2, Alg.1 contemplates using registers to pass values between cores, such as between controller and minion routines in Alg.1, see, e.g., Function ParaSa: Minion(schedule[id-1],recv()), Function Ctl … scatter(Par::Upd(delta-sub-t, l_act, l_state)) showing scatter broadcast and receiving of various configuration values to the minion tasks, such as for determining actions and updating based on the passed parameters, hence, registers storing such configuration values); and
a range of memory addresses allocated to storing formatting data for the RL routine (Simpson fig.1b, §4.1 contemplates memory hierarchies for storing various data, with the CLS memory segment set aside for config (state config, tile config, policy data) and atomic writeback count and values (fig.1b), hence, storing formatting data for state, policy, tiles in that memory range).
For claim 11, Simpson discloses the unit of claim 9, as described above. Simpson further discloses: wherein the accelerator circuit is further to
cause, on behalf of the host processing device, the RL routine top perform further reinforcement learning via updates to one or more of the configuration values and responsive to inferences made based on the RL-related action (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism of fig.3a-b, hence, based on activity rewards (See Alg.1: lookup_reward_from_key), the configuration value tiles are updated); and
continue to process additional network packets, received at the particular port and using one or more updated configuration value, through the RL routine on behalf of the host processing device (fig.3a-b, Alg.1: rewards, policy, and updating continues using updated tiling configurations.
For claim 17, Simpson discloses: a method comprising:
detecting, by accelerator circuitry of a network interface device (§4.1-2 gives accelerator processor circuit overview) operatively coupled with a host processing device (§4.1-4.2), a plurality of network packets that are received (§5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense); and
executing a reinforcement learning (RL) routine, on behalf of the host processing device (§4.1-2 gives overview of NIC and host architecture including blade servers, with §1 ¶3 (“It is important …” contemplating the importance of offloading learning and inference to NIC’s rather than hosts), by the accelerator circuitry (ibid: the RL routine is executed), using configuration values associated with a set of RL-related parameters (§3.5 contemplates reconfiguration via a host computer via control packets during runtime) and in response to detecting the plurality of network packets (§5.1), and
wherein executing the RL routine comprises updating a data structure storing action values for discrete states (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) and employing observation information derived from or associated with the network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment levels, dropping packets, changing rate limits) to train the RL routine to perform inferencing on further network packets of the plurality of network packets (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism modes of fig.3a-b; hence, inference is performed on future network packets).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose: wherein the packets are received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
For claim 18, Simpson modified by Viegas discloses the unit of claim 17, as described above. Simpson modified by Viegas further discloses: wherein the network packets are further processed through a second RL routine based on a particular criterion comprising containing a particular identifier comprising one of a destination address or a fixed byte portion of each network packet (§4.2 last ¶ discloses various extracted features including IP address (see “DDOS” Table 1, §IV); see also §2.3 ¶2 contemplating taking header packet data as input, such as for a preference or priority list policy, the packet header data containing packet identifiers such as source, destination, the data being a fixed byte portion of a packet, the data being used to track flow state such as described in §5.1 ¶1, with §5.2 ¶2, §5.1 ¶1-3 contemplating deployments across a network).
Claim(s) 3-5, 7-8, 12-14, 16, 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016) in view of Wybenga (US 20050249206 A1).
For claim 3, Simpson modified by Viegas discloses the unit of claim 1, as described above. Simpson modified by Viegas further discloses: wherein, to execute the RL routine, the accelerator circuitry is further to:
derive the observation information from the network packet, the observation information being associated with an RL-related parameter of the set of RL-related parameters (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
update a counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
update, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
update, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas does not disclose: wherein the counter is hardware, wherein the stored is in a hardware register.
Wybenga discloses: wherein the counter is hardware, wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 4, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Viegas modified by Wybenga further discloses: wherein the RL routine comprises a temporal difference (TD) algorithm that is to compare outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 5, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Simpson DDOS further discloses: an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value based on an RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states), wherein the accelerator circuitry is further to:
iteratively update the cumulative state value across iterations of the RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determine, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update algorithm, hence, determining subsequent configuration values (i.e., tiles values and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further update the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 7, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Viegas modified by Wybenga further discloses: the hardware counter and the hardware register coupled with the accelerator circuitry (Simpson fig.1 shows coupling of processor and memory within the chip), and wherein the memory comprises:
one or more software registers to store the configuration values (Simpson fig.1b, §3.4 ¶2, Alg.1 contemplates using registers to pass values between cores, such as between controller and minion routines in Alg.1, see, e.g., Function ParaSa: Minion(schedule[id-1],recv()), Function Ctl … scatter(Par::Upd(delta-sub-t, l_act, l_state)) showing scatter broadcast and receiving of various configuration values to the minion tasks, such as for determining actions and updating based on the passed parameters, hence, registers encoding such data storing such configuration values); and
a range of memory addresses allocated to storing formatting data for the RL routine (Simpson fig.1b, §4.1 contemplates memory hierarchies for storing various data, with the CLS memory segment set aside for config (state config, tile config, policy data) and atomic writeback count and values (fig.1b), hence, storing formatting data for state, policy, tiles in that memory range).
For claim 8, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 3, as described above. Simpson further discloses: an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters), wherein the accelerator circuitry is coupled with a host processing device (§3.5 contemplates host updating of NIC accelerators via control packets), the accelerator circuitry further to:
detect, based on an interrupt request received from the host processing device, a context switch to an application associated with the RL-related parameter (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
deploy a configuration file received from the host processing device to program the memory and the accelerator circuitry to perform reinforcement learning using the RL routine, wherein to deploy the configuration file (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine), the accelerator circuitry is further to:
load, into the memory, initial configuration values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine); and
load, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and loaded onto the memory cache shown in fig.1a-b).
For claim 12, Simpson modified by Viegas discloses the unit of claim 9, as described above. Simpson further discloses: wherein the NIC further comprises:
wherein, to execute the RL routine, the accelerator circuit is further to:
derive the observation information from the network packet, the observation information being associated with an RL-related parameter of the set of RL-related parameters (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
update the counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
update, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
update, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas does not disclose: a hardware counter coupled with the accelerator unit, a hardware register coupled to the accelerator circuit wherein the stored is in a hardware register, wherein the stored is in a hardware register.
Wybenga discloses: a hardware counter coupled with the accelerator unit, a hardware register coupled to the accelerator circuit wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, the counters and registers being read and accessed by the coupled processor, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 13, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 12, as described above. Simpson further discloses: wherein, to execute the RL routine, the accelerator circuit is to perform a temporal difference (TD) algorithm by comparing outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 14, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 12, as described above. Simpson further discloses: wherein the NIC further comprises an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value based on an RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states), wherein the accelerator circuit is further to:
iteratively update the cumulative state value across iterations of the RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determine, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update calculations eq.1-2, hence, determining subsequent configuration values (e.g., tiles and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further update the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 16, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 12, as described above. Simpson modified by Viegas modified by Wybenga further discloses: wherein the NIC further comprises an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters), wherein the accelerator circuit is further to:
detect, based on the interrupt request received from the host processing device, a context switch to the application (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
deploy the configuration file received from the host processing device to program the memory and the accelerator circuit to perform reinforcement learning using the RL routine, wherein to deploy the configuration file, the accelerator circuit is further to (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine):
load, into the memory, initial configuration values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine);
load, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and loaded onto the memory cache shown in fig.1a-b); and
identify an address that points to the hardware register (Wybenga fig.5, 0099: addresses are identified for access of counter and register values).
For claim 19, Simpson modified by Viegas discloses the unit of claim 17, as described above. Simpson further discloses: wherein executing the RL routine further comprises:
deriving the observation information from the network packet, the observation information being associated with an RL-related parameter (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
updating a counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
updating, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
updating, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas does not disclose: wherein the counter is a hardware counter, wherein the stored is in a hardware register.
Wybenga discloses: wherein the counter is hardware counter, wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, the counters and registers being read and accessed by the coupled processor, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 20, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: wherein the RL routine comprises a temporal difference (TD) algorithm that is to compare outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 21, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: storing, within on-chip cache of the network interface device (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache), a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states);
iteratively updating the cumulative state value across iterations of the RL routine (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determining, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update calculations eq.1-2, hence, determining subsequent configuration values (e.g., tiles and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further updating the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 22, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: storing, within on-chip cache of the network interface device (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache), a data structure in which configuration value and action value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters, Simpson Revisit §3.2 ¶1: updated action values are executed and sent into the environment);
detecting, based on an interrupt request received from a host processing device, a context switch to an application associated with the RL-related parameter (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
executing a configuration file received from the host processing device to program a memory and the accelerator circuitry to perform reinforcement learning using the RL routine, wherein to deploy the configuration file (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine), the method further comprising:
loading, into the memory, initial configuration values and action values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data, hence, values for actions to be taken, is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine); and
loading, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values and action values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and associated actions, hence, action values, and loaded onto the memory cache shown in fig.1a-b).
Claim(s) 6, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation by reference of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016) in view of Wybenga (US 20050249206 A1) in view of Wang (CN 110581808 A).
For claim 6, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 5, as described above. Simpson modified by Viegas modified by Wybenga does not disclose the limitations of claim 6.
Wang discloses: wherein the accelerator circuitry is further to, in response to detecting the cumulative state value satisfying a target end value based on the algorithm that governs updates to the data structure, perform a final update of the configuration value associated with the RL-related parameter based on a final cumulative state value (p.5 ¶3, p.7 2nd ¶ from bottom contemplates tracking a cumulative reward state value and determining whether that value satisfies a threshold value based on the algorithm and performing a final update corresponding to generating the control model associated with the various RL parameters based on final reward value).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the reward threshold value parameter of Wang. Both concern the art reinforcement learning in traffic or congestion control, and the incorporation would have, according to Wang, allowing a termination condition based on reward value (p.5 ¶3, p.7 2nd ¶ from bottom), hence, better improving network performance (abstract).
For claim 15, Simpson modified by Viegas modified by Wybenga discloses the unit of claim 14, as described above. Simpson modified by Viegas modified by Wybenga does not disclose the limitations of claim 15.
Wang discloses: wherein the accelerator circuit is further to, in response to detecting the cumulative state value satisfying a target based on the algorithm that governs updates to the data structure, perform a final update of the configuration value associated with the RL-related parameter based on a final cumulative state value. (p.5 ¶3, p.7 2nd ¶ from bottom contemplates tracking a cumulative reward state value and determining whether that value satisfies a threshold value based on the algorithm and performing a final update corresponding to generating the control model associated with the various RL parameters based on final reward value).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the reward threshold value parameter of Wang. Both concern the art reinforcement learning in traffic or congestion control, and the incorporation would have, according to Wang, allowing a termination condition based on reward value (p.5 ¶3, p.7 2nd ¶ from bottom), hence, better improving network performance (abstract).
Response to Arguments
Applicant’s arguments have been fully considered. In the remarks, Applicant argues:
1. The art of record does not disclose the amended claims, because the art of record, and Simpson Revisit and Simpson DDoS in particular, do not disclose the newly added limitations directed to updating a data structure storing action values for discrete states, as both Simpson references use tile coding with linear function approximation – not a data structure storing action values for discrete states.
Examiner respectfully disagrees. Tile coding is generally disclosed in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value.
Does tile coding “update a data structure storing action values for discrete states”? Examiner submits that it indeed does do so in various ways. We can map w to the data structure storing action values (i.e., expected reward components given a state and an action). The set of states are encoded in various ways, but most directly via the features themselves, such as shown in p.107: table 1. Many of these values are discrete, including IP, packets in/out, mean in/out packet size, etc.; hence, s can be considered to be discrete prior to any particular tile coding. Furthermore, as x basically acts like a binary selector on the values of w, x(s) itself, keyed to a particular state, constitutes a discrete state representation that is used to select over w.
However, Examiner acknowledges that tile coding is not equivalent to Q-tables (such as in Specs 0024-27), which is a form of storing a one-to-one correspondence between a discrete state, action, and reward, would overcome the art of record.
2. Simpson Revisit GitHub states … In tile coding, policy is represented by sets of tiles covering dimensions of the input state with overlapping tiles, and action values are approximated using weight vectors – not stored directly for discrete states.
For tile coding, action values are calculated for each state based on a Boolean selector x(s,a) selecting over the set of tiles w. Examiner submits that a plurality of action values are stored for each stats, or, if as above, if we take active elements of x to be a set of states (the discrete set of x corresponding to each real state), then a unique action value is stored for each state x. However, in both cases action values, values – the elements of w in eq.1 -- corresponding to particular actions, are stored for discrete states.
3. Simpson DDOS teaches away from storing action values for discrete states, as exact tabular representations are infeasible for higher dimensional states.
Examiner agrees that Simpson teaches away from “exact tabular representations”, e.g., a Q-table, but the claim language can be fulfilled even without exact tabular representations. Furthermore, Examiner has carefully reviewed the Specifications, including cited portions 0002-3, 0035-39 and did not find any indication of any tabular representations being described.
4. A fundamental difference exists between tile coding and discrete state / action values, in that action values are approximated via summation of weight vectors, which is different from the action values storing and updated for discrete states, such as a Q-table described in Applicant’s Specifications.
Examiner respectfully disagrees, as described above, that tile codings do not contain discrete state and associated data. Hence, tile coding fulfills the claim language of a “data structure storing action values for discrete states”, as described above. Examiner acknowledges that amending to explicitly recite Q-tables, which are data structures storing a unique action for each state-action pair, would overcome the art of record.
5. Improper motivation: Simpson Revisit and Viegas are directed to different fields; notably, Viegas does not teach reinforcement learning at all and is not focused on the networking optimization features of Simpson.
Examiner respectfully disagrees. Both concern the art of network control, including network security, intrusion detection, DoS attacks (Simpson Revisit §3.1 ¶2, §3.2 ¶3, §4.2 last ¶, etc.; Viegas: abstract, §2.1) via monitoring of network parameters via machine learning. Hence, Viegas, being drawn from the same art of machine-learning techniques for network intrusion, is relied upon for disclosing the additional consideration of port numbers as a feature. A POSITA would have been motivated to monitor these additional features in order to better detect network flow anomalies associated with these features, see §2.1.
6. Simpson Revisit GitHub addresses RL learning to enable adaptive network control, while Viegas addresses the problem of energy-efficient intrusion detection via supervised classification. There is no teaching, suggestion, or motivation, and one of ordinary skill in the art would not look to Viegas to modify Simpsons reinforcement learning system.
Examiner respectfully disagrees. Viegas is relied on merely to further consider the port number in the set of input features, such as to enable better network anomaly detection and classification, see §2.1. As all machine learning models, whether supervised or reinforcement learning, can operate over a wide range of feature data, no significant modifications would be needed to ingest the additional data being taught by Viegas and a POSITA would find such a combination a predictable result. A POSITA would further be motivated to make such a combination for the reasons given above.
7. In response to the advisory action, Examiner oversimplifies the claim and arguments; in particular, action values are not stored for discrete states.
Examiner respectfully disagrees. Many action values are stored for each discrete state, for example, depending on the action, on the tile etc. See Simpson DDOS §II.B as described above.
8. In response to the advisory action, Examiner has not addressed the teaching awayh argument in reference to Simpson DDoS, which has been incorporated into the Simpson Revisit GitHub reference.
As described above, both Simpson references and the Viegas reference concern the art of network intrusion or anomaly detection via machine learning. A POSITA would be motivated to combine for the reasons previously given.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Rai (US 20230284254 A1) discloses the use of Q-learning in a radio communications context.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
/LIANG LI/
Primary examiner AU 2143