DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to pending claims 1-22 filed 6/22/2026.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 9-11, 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation by reference of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016) in view of in view of Araqi ("Joint Channel Assignment and Multicast Routing in Multi-Channel Multi-Radio Wireless Mesh Networks Based on Q-Learning", published 2019).
For claim 1, Simpson discloses: a network interface device (§3 ¶1-2, figs.1-3 contemplates implementation on SmartNIC devices, see also §4.1-2 giving hardware overview) comprising:
a memory to store configuration values (§4.1-2) associated with a reinforcement learning (RL) routine and a set of RL-related parameters of the RL routine (fig.1 shows RL routine and associated parameters, e.g., state, reward, configuration data);
packet processing circuitry to receive network packets (§4.1-2, fig.1 shows incoming packets received by the P4 pipeline); and
accelerator circuitry coupled to the memory, the packet processing circuitry (§4.1-2 gives accelerator processor circuit overview), and a host processing device (§4.2), wherein the accelerator circuitry is to:
detect a network packet (§2.3 ¶2, §5.1 ¶1 contemplates use of algorithms for detecting particular packet header state data in association with actions, such as with various flow preference levels, such as for DDoS defense; see also §4.2 ¶2-3 describing determining packet state in the context of implementing DDoS RL control application, the state including various tilings on a 20-dim state vector and 10 possible actions corresponding to limiting traffic flow via incorporation of Simpson’s earlier work “Per-Host DDoS Mitigation by Direct-Control Reinforcement Learning”, particularly §III.C “Feature Space”: Table 1 describing the feature space including IP address, §IV: “Rethinking the State Space” describing the rationale for incorporating various features such as IP address; hence, a packet and its corresponding state is detected) ; and
execute the RL routine, using the configuration values and in response to detecting the network packet (ibid: the RL routine is executed based on the detected packet and state), to update a data structure (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) and to employ observation information derived from or associated with the network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment / preference levels, dropping packets, changing rate limits)
cause, on behalf of the host processing device, the RL routine to perform further reinforcement learning via updates to one or more of the configuration values and action values and responsive to inferences made based on the RL-related action (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism modes of fig.3a-b; hence, based on activity rewards (See Alg.1: lookup_reward_from_key), the configuration value tiles are updated; see §3.2 ¶1 disclosing the updating of action values, i.e., the determining of a current action and sending it out into the environment); and
continue to process additional network packets using one or more updated configuration values and action values, through the RL routine on behalf of the host processing device (fig.3a-b, Alg.1: rewards, policy, and updating continues using updated tiling configurations; §3.2 ¶1 disclosing updating and executing action values).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose:
wherein the packet is received at a particular port of a plurality of ports of the network interface device;
wherein the detecting detects received at a particular port;
wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
Simpson modified by Viegas does not disclose: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Araqi discloses: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value (§IV discloses proposed algorithm, including updating client tables (step 15), the actions being chosen at based on the policy (step 10, see §II ¶2-3 for a description of storing action state pairs, hence, storing actions for discrete states), the agents residing on the NICs (step 4)).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Simpson by incorporating the q-table technique of Araqi. Both concern the art of reinforcement learning on NICs, and the incorporation would have, according to Araqi, improve network performance via Q-learning on NICs (§IV).
For claim 2, Simpson modified by Viegas modified by Araqi discloses the unit of claim 1, as described above. Simpson modified by Viegas further discloses: wherein the network packets are further processed through a second RL routine based on a particular criterion comprising containing a particular identifier comprising one of a destination address or a fixed byte portion of each network packet (§4.2 last ¶ discloses various extracted features including IP address (see “DDOS” Table 1, §IV); see also §2.3 ¶2 contemplating taking header packet data as input, such as for a preference or priority list policy, the packet header data containing packet identifiers such as source, destination, the data being a fixed byte portion of a packet, the data being used to track flow state such as described in §5.1 ¶1, with §5.2 ¶2, §5.1 ¶1-3 contemplating deployments across a network).
For claim 9, Simpson discloses: a data processing unit comprising:
a network interface card (NIC) (§3 ¶1-2, figs.1-3 contemplates implementation on SmartNIC devices, see also §4.1 giving hardware overview) comprising:
a memory to store configuration values (§4.1) associated with a reinforcement learning (RL) routine and a set of RL-related parameters for implementing the RL routine (fig.1 shows RL routine and associated parameters, e.g., state, reward, configuration data);
a network interface to receive network packets (§4.1, fig.1 shows incoming packets received by the P4 pipeline); and
an accelerator circuit coupled to the memory and the network interface (§4.1-4.2 gives accelerator processor circuit overview), the accelerator circuit to iteratively:
detect a network packet that is received (§5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense); and
execute the RL routine, using the configuration values and in response to detecting the network packet (ibid: the RL routine is executed), to update a data structure (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) to employ observation information derived from or associated with a network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment levels, dropping packets, changing rate limits; §5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense, with §4.2 describing the packet in the context of total state including various tiling on a 20-dim state vector and 10 possible actions corresponding to limiting traffic flow via incorporation of Simpson’s earlier work “Per-Host DDoS Mitigation by Direct-Control Reinforcement Learning”, particularly §III.C “Feature Space”: Table 1 describing the feature space, e.g., including IP address, §IV: “Rethinking the State Space” describing the rationale for incorporating various features such as IP address; hence, a packet and its corresponding state is detected and observed to perform RL actions); and
a host processing device coupled with the NIC and the accelerator circuit (§3.5 contemplates reconfiguration via a host computer via control packets during runtime), the host processing device to:
send an interrupt request to an RL configuration interface of the accelerator circuit to cause a context switch of the accelerator circuit to an application associated with the RL routine (ibid: sending learning parameters, policy design changes such as for a new policy for another problem, hence, interrupting a currently executing policy for another problem, hence, causing a context change; these learning parameters being received by the configuration interface of the NIC, such as via UDP and reserved values (§3.5 ¶1)); and
send a configuration file to the NIC, the configuration file to cause the accelerator circuit to program the configuration values, the action values, and associated formatting data into the memory (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, action values, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose: wherein the packet is received at a particular port of a plurality of a plurality of ports of the NIC; wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
Simpson modified by Viegas does not disclose: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Araqi discloses: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value (§IV discloses proposed algorithm, including updating client tables (step 15), the actions being chosen at based on the policy (step 10, see §II ¶2-3 for a description of storing action state pairs, hence, storing actions for discrete states), the agents residing on the NICs (step 4)).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Simpson by incorporating the q-table technique of Araqi. Both concern the art of reinforcement learning on NICs, and the incorporation would have, according to Araqi, improve network performance via Q-learning on NICs (§IV).
For claim 10, Simpson modified by Viegas modified by Araqi discloses the unit of claim 9, as described above. Simpson further discloses: wherein the memory comprises:
one or more software registers to store the configuration values (Simpson fig.1b, §3.4 ¶2, Alg.1 contemplates using registers to pass values between cores, such as between controller and minion routines in Alg.1, see, e.g., Function ParaSa: Minion(schedule[id-1],recv()), Function Ctl … scatter(Par::Upd(delta-sub-t, l_act, l_state)) showing scatter broadcast and receiving of various configuration values to the minion tasks, such as for determining actions and updating based on the passed parameters, hence, registers storing such configuration values); and
a range of memory addresses allocated to storing formatting data for the RL routine (Simpson fig.1b, §4.1 contemplates memory hierarchies for storing various data, with the CLS memory segment set aside for config (state config, tile config, policy data) and atomic writeback count and values (fig.1b), hence, storing formatting data for state, policy, tiles in that memory range).
For claim 11, Simpson modified by Viegas modified by Araqi discloses the unit of claim 9, as described above. Simpson further discloses: wherein the accelerator circuit is further to
cause, on behalf of the host processing device, the RL routine top perform further reinforcement learning via updates to one or more of the configuration values and responsive to inferences made based on the RL-related action (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism of fig.3a-b, hence, based on activity rewards (See Alg.1: lookup_reward_from_key), the configuration value tiles are updated); and
continue to process additional network packets, received at the particular port and using one or more updated configuration value, through the RL routine on behalf of the host processing device (fig.3a-b, Alg.1: rewards, policy, and updating continues using updated tiling configurations.
For claim 17, Simpson discloses: a method comprising:
detecting, by accelerator circuitry of a network interface device (§4.1-2 gives accelerator processor circuit overview) operatively coupled with a host processing device (§4.1-4.2), a plurality of network packets that are received (§5.1 ¶1 contemplates use of algorithms for detecting particular packet flows and association with various punishment levels for DDoS defense); and
executing a reinforcement learning (RL) routine, on behalf of the host processing device (§4.1-2 gives overview of NIC and host architecture including blade servers, with §1 ¶3 (“It is important …” contemplating the importance of offloading learning and inference to NIC’s rather than hosts), by the accelerator circuitry (ibid: the RL routine is executed), using configuration values associated with a set of RL-related parameters (§3.5 contemplates reconfiguration via a host computer via control packets during runtime) and in response to detecting the plurality of network packets (§5.1), and
wherein executing the RL routine comprises updating a data structure (Tile coding implementation, as referenced by Simpson §4.2 ¶2-3, is described in Simpson DDoS §II.B (p.104-105), particularly figs. 1-2 and eq.1. Eq.1 goes over the calculating of q-hat, the current expected reward value, given current state s and action a. Each dimension is binned, multiply, into an overlapping set of discrete tiles. For each action a, and given the current state s, the sparse Boolean matrix x(s, a) acts like a large selector over the entries of w, where w encodes the values of the various tiles (see figs. 1-2), with x selecting a values of w to sum to generate the expected reward q-hat for the particular action. Finally, depending on the algorithm, e.g., greedy algorithm, an action is chosen based on the expected reward q-hat value. Hence, the set of states are themselves discrete , such as shown in Simpson DDOS p.107:table 1, and furthermore, find translation to particular discrete binned states for each tile, as well as to particular discrete representations of the Boolean spares matrix x, all of which would be discrete states, the associated action values of which are updated in the matrix w) and employing observation information derived from or associated with the network packet to perform an RL-related action (ibid: associated actions are performed, such as adjusting punishment levels, dropping packets, changing rate limits) to train the RL routine to perform inferencing on further network packets of the plurality of network packets (Alg.1 (p.6) discloses minion tasks tasked with updating (Par::Upd message) and policy inference (Par::Act message) on the tasks, these updates handled via the parallelism modes of fig.3a-b; hence, inference is performed on future network packets).
Although Simpson contemplates the use of various packet features including IP address in the feature space (“Revisiting” §4.2 Last Paragraph, “Per-Host DDoS” §III.C Table 1, §IV: Source IP Address), he does not contemplate explicitly include port numbers. Hence, Simpson does not disclose: wherein the packets are received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port; wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Viegas discloses: wherein the packet is received at a particular port of a plurality of ports of the network interface device; wherein the detecting detects received at a particular port (§4.1 “Table-based extraction method” ¶1-6 (“For service-based features…”: contemplates tracking and indexing port numbers for traffic exchanged in order to detect network anomalies, such as to track services (¶1), hence, a packet being received at a particular port, i.e., processed and handled according to its incoming port number).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson, incorporating the feature and action state of Simpson DDOS, by incorporating the port number tracking of Viegas. Both concern the art of network anomaly detection via hardware implementations, and the incorporation would have, according to Viegas, allowed tracking of features related to network services for network traffic in order to detect network flow anomalies (§4.1 ¶1).
Simpson modified by Viegas does not disclose: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value.
Araqi discloses: wherein the data structure is storing, for each of one or more discrete states, a corresponding action value (§IV discloses proposed algorithm, including updating client tables (step 15), the actions being chosen at based on the policy (step 10, see §II ¶2-3 for a description of storing action state pairs, hence, storing actions for discrete states), the agents residing on the NICs (step 4)).
It would have been obvious before the effective filing date to one of ordinary skill in the art to modify the method of Simpson by incorporating the q-table technique of Araqi. Both concern the art of reinforcement learning on NICs, and the incorporation would have, according to Araqi, improve network performance via Q-learning on NICs (§IV).
For claim 18, Simpson modified by Viegas modified by Araqi discloses the unit of claim 17, as described above. Simpson modified by Viegas further discloses: wherein the network packets are further processed through a second RL routine based on a particular criterion comprising containing a particular identifier comprising one of a destination address or a fixed byte portion of each network packet (§4.2 last ¶ discloses various extracted features including IP address (see “DDOS” Table 1, §IV); see also §2.3 ¶2 contemplating taking header packet data as input, such as for a preference or priority list policy, the packet header data containing packet identifiers such as source, destination, the data being a fixed byte portion of a packet, the data being used to track flow state such as described in §5.1 ¶1, with §5.2 ¶2, §5.1 ¶1-3 contemplating deployments across a network).
Claim(s) 3-5, 7-8, 12-14, 16, 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016) in view of Araqi ("Joint Channel Assignment and Multicast Routing in Multi-Channel Multi-Radio Wireless Mesh Networks Based on Q-Learning", published 2019) in view of Wybenga (US 20050249206 A1).
For claim 3, Simpson modified by Viegas modified by Araqi discloses the unit of claim 1, as described above. Simpson modified by Viegas further discloses: wherein, to execute the RL routine, the accelerator circuitry is further to:
derive the observation information from the network packet, the observation information being associated with an RL-related parameter of the set of RL-related parameters (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
update a counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
update, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
update, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas modified by Araqi does not disclose: wherein the counter is hardware, wherein the stored is in a hardware register.
Wybenga discloses: wherein the counter is hardware, wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas modified by Araqi by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 4, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Viegas modified by Wybenga further discloses: wherein the RL routine comprises a temporal difference (TD) algorithm that is to compare outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 5, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Simpson DDOS further discloses: an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value based on an RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states), wherein the accelerator circuitry is further to:
iteratively update the cumulative state value across iterations of the RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determine, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update algorithm, hence, determining subsequent configuration values (i.e., tiles values and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further update the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 7, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 3, as described above. Simpson modified by Viegas modified by Wybenga further discloses: the hardware counter and the hardware register coupled with the accelerator circuitry (Simpson fig.1 shows coupling of processor and memory within the chip), and wherein the memory comprises:
one or more software registers to store the configuration values (Simpson fig.1b, §3.4 ¶2, Alg.1 contemplates using registers to pass values between cores, such as between controller and minion routines in Alg.1, see, e.g., Function ParaSa: Minion(schedule[id-1],recv()), Function Ctl … scatter(Par::Upd(delta-sub-t, l_act, l_state)) showing scatter broadcast and receiving of various configuration values to the minion tasks, such as for determining actions and updating based on the passed parameters, hence, registers encoding such data storing such configuration values); and
a range of memory addresses allocated to storing formatting data for the RL routine (Simpson fig.1b, §4.1 contemplates memory hierarchies for storing various data, with the CLS memory segment set aside for config (state config, tile config, policy data) and atomic writeback count and values (fig.1b), hence, storing formatting data for state, policy, tiles in that memory range).
For claim 8, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 3, as described above. Simpson further discloses: an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters), wherein the accelerator circuitry is coupled with a host processing device (§3.5 contemplates host updating of NIC accelerators via control packets), the accelerator circuitry further to:
detect, based on an interrupt request received from the host processing device, a context switch to an application associated with the RL-related parameter (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
deploy a configuration file received from the host processing device to program the memory and the accelerator circuitry to perform reinforcement learning using the RL routine, wherein to deploy the configuration file (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine), the accelerator circuitry is further to:
load, into the memory, initial configuration values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine); and
load, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and loaded onto the memory cache shown in fig.1a-b).
For claim 12, Simpson modified by Araqi modified by Viegas discloses the unit of claim 9, as described above. Simpson further discloses: wherein the NIC further comprises:
wherein, to execute the RL routine, the accelerator circuit is further to:
derive the observation information from the network packet, the observation information being associated with an RL-related parameter of the set of RL-related parameters (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
update the counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
update, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
update, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas does not disclose: a hardware counter coupled with the accelerator unit, a hardware register coupled to the accelerator circuit wherein the stored is in a hardware register, wherein the stored is in a hardware register.
Wybenga discloses: a hardware counter coupled with the accelerator unit, a hardware register coupled to the accelerator circuit wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, the counters and registers being read and accessed by the coupled processor, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 13, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 12, as described above. Simpson further discloses: wherein, to execute the RL routine, the accelerator circuit is to perform a temporal difference (TD) algorithm by comparing outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 14, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 12, as described above. Simpson further discloses: wherein the NIC further comprises an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value based on an RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states), wherein the accelerator circuit is further to:
iteratively update the cumulative state value across iterations of the RL algorithm (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determine, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update calculations eq.1-2, hence, determining subsequent configuration values (e.g., tiles and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further update the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 16, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 12, as described above. Simpson modified by Viegas modified by Wybenga further discloses: wherein the NIC further comprises an on-chip cache to store (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache) a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters), wherein the accelerator circuit is further to:
detect, based on the interrupt request received from the host processing device, a context switch to the application (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
deploy the configuration file received from the host processing device to program the memory and the accelerator circuit to perform reinforcement learning using the RL routine, wherein to deploy the configuration file, the accelerator circuit is further to (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine):
load, into the memory, initial configuration values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine);
load, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and loaded onto the memory cache shown in fig.1a-b); and
identify an address that points to the hardware register (Wybenga fig.5, 0099: addresses are identified for access of counter and register values).
For claim 19, Simpson modified by Viegas modified by Araqi discloses the unit of claim 17, as described above. Simpson further discloses: wherein executing the RL routine further comprises:
deriving the observation information from the network packet, the observation information being associated with an RL-related parameter (Simpson “Revisit” §2.3 ¶2: associating header data with observation state information; §4.2 ¶2: associating observation state vectors found in real-world DDoS of Simpson “DDoS”; Simpson “DDoS” §III.C “Feature Space” discloses various global and per-flow observation data associate with a packet for a RL algorithm, with §IV giving a further elaboration);
updating a counter corresponding to the observation information (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window);
updating, based on a value of the counter, a cumulative state value stored (Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) discloses using tile coding to track policy, hence, updating cumulative tiles determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters); and
updating, based on the cumulative state value, a configuration value for the RL-related parameter (Simpson “Revisit” Revisit” §3.2 ¶1, Alg.1 (p.6); Simpson “DDoS”: §II.B eq.2, as above: configuration values, corresponding to update values for the cumulative state values, are updated based on current and subsequent cumulative state values).
Simpson modified by Viegas does not disclose: wherein the counter is a hardware counter, wherein the stored is in a hardware register.
Wybenga discloses: wherein the counter is hardware counter, wherein the stored is in a hardware register (Wybenga fig.5, 0099: use of a hardware counter and hardware registers such as overflow, the counters and registers being read and accessed by the coupled processor, furthermore, combination with Simpson reading the packet count value to a on chip hardware register for further operations (see Simpson §4.1), such as determining per-window count, arithmetic operations, message passing (Simpson §3.4), etc.).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the hardware counter and registers of Wybenga. Both concern the network traffic monitoring, and the incorporation would have, according to Wybenga, allow improved packet counters and adders with improved performance and accessibility (0010-11).
For claim 20, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: wherein the RL routine comprises a temporal difference (TD) algorithm that is to compare outcomes of temporally-successive predictions, and wherein the update to the configuration value includes a reward prediction (§3.2: Sarsa is a TD algorithm that compares temporally-successive predictions by subtracting the value of the current state with the successive state, and by incorporating a learning rate hyperparameter with a reward rate of the current state).
For claim 21, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: storing, within on-chip cache of the network interface device (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache), a data structure in which configuration value updates correspond to a particular value range for the cumulative state value (Simpson “Revisit” §4.2; Alg.1 (p.6); “DDOS” §II.B, figs.1-2: updating of various tile data structures corresponding to policies, each tile corresponding to particular value range for cumulative policies states);
iteratively updating the cumulative state value across iterations of the RL routine (Simpson “Revisit” §4.2; Alg.1 (p.6); DDOS §II.B, figs.1-2: cumulative policy values are updated iteratively according to described algorithm, equations, during RL learning training);
determining, from the data structure, a subsequent configuration value that corresponds to the updated cumulative state value (ibid: cumulative states are updated according to update calculations eq.1-2, hence, determining subsequent configuration values (e.g., tiles and multipliers for subsequent timesteps) corresponding to the updated cumulative values); and
further updating the configuration value to the subsequent configuration value for a subsequent iteration of the RL routine (ibid: update calculations and corresponding subsequent configuration values are likewise determined for future iterations).
For claim 22, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 19, as described above. Simpson further discloses: storing, within on-chip cache of the network interface device (Simpson fig.1a-b shows policy being stored across Local CLS, CTM, and IMEM, see also §4.1 ¶1 describing the use of the various policies, these memories being on-chip local access locations that save access cost compared to more distant and larger memory, hence, on-chip cache), a data structure in which configuration value and action value updates correspond to a particular value range for the cumulative state value (As referenced by Simpson “Revisit” §4.2, Simpson “DDoS”: §III.C ¶4, §IV ¶9: “(Per-window) packet count” discloses updating counters indicating packets associated with a flow lifetime or window, with Simpson “Revisit” §3.2 ¶1, Alg.1 (p.6) disclosing using tile coding to track policy, hence, updating configuration values determining policy based on counter as state value based on reward feedback; see also tile coding as disclosed by Simpson “DDoS” §II.B, figs.1-2: various cumulative values of the tiles or policies are updated based on the counters, Simpson Revisit §3.2 ¶1: updated action values are executed and sent into the environment);
detecting, based on an interrupt request received from a host processing device, a context switch to an application associated with the RL-related parameter (§3.5: change of policy design and learning parameters, such as to interrupt a current phase or to adapt to a new problem); and
executing a configuration file received from the host processing device to program a memory and the accelerator circuitry to perform reinforcement learning using the RL routine, wherein to deploy the configuration file (ibid: control packets constitute formatted data, hence, file data, causing parameter changes and policy design changes, hence, causing the processor accelerator circuit on the NIC to program the respective configuration values and associated data into memory for operating the RL routine), the method further comprising:
loading, into the memory, initial configuration values and action values for the set of RL-related parameters and associated formatting data received from the host processing device (ibid: the initial parameters such as policy data, hence, values for actions to be taken, is loaded, see also §4.2 ¶3 describing policy dimension data, hence, formatting data is loaded and initial configurations are loaded for the RL routine); and
loading, into the on-chip cache, the data structure from the memory that includes past cumulative state values and associated updates to the configuration values and action values (§3.5: loading policy data from past, pre-trained models including past policy values based on past training updates and associated actions, hence, action values, and loaded onto the memory cache shown in fig.1a-b).
Claim(s) 6, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Simpson Revisit Github ("Revisiting the Classics: Online RL in the Programmable Dataplane", published 7/10/2021) with incorporation by reference of Simpson DDoS ("Per-host DDoS mitigation by direct-control reinforcement learning", published 2019) in view of Viegas ("Towards an energy-efficient anomaly-based intrusion detection engine for embedded systems", published 2016) in view of Araqi ("Joint Channel Assignment and Multicast Routing in Multi-Channel Multi-Radio Wireless Mesh Networks Based on Q-Learning", published 2019) in view of Wybenga (US 20050249206 A1) in view of Wang (CN 110581808 A).
For claim 6, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 5, as described above. Simpson modified by Viegas modified by Araqi modified by Wybenga does not disclose the limitations of claim 6.
Wang discloses: wherein the accelerator circuitry is further to, in response to detecting the cumulative state value satisfying a target end value based on the algorithm that governs updates to the data structure, perform a final update of the configuration value associated with the RL-related parameter based on a final cumulative state value (p.5 ¶3, p.7 2nd ¶ from bottom contemplates tracking a cumulative reward state value and determining whether that value satisfies a threshold value based on the algorithm and performing a final update corresponding to generating the control model associated with the various RL parameters based on final reward value).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the reward threshold value parameter of Wang. Both concern the art reinforcement learning in traffic or congestion control, and the incorporation would have, according to Wang, allowing a termination condition based on reward value (p.5 ¶3, p.7 2nd ¶ from bottom), hence, better improving network performance (abstract).
For claim 15, Simpson modified by Viegas modified by Araqi modified by Wybenga discloses the unit of claim 14, as described above. Simpson modified by Viegas modified by Araqi modified by Wybenga does not disclose the limitations of claim 15.
Wang discloses: wherein the accelerator circuit is further to, in response to detecting the cumulative state value satisfying a target based on the algorithm that governs updates to the data structure, perform a final update of the configuration value associated with the RL-related parameter based on a final cumulative state value. (p.5 ¶3, p.7 2nd ¶ from bottom contemplates tracking a cumulative reward state value and determining whether that value satisfies a threshold value based on the algorithm and performing a final update corresponding to generating the control model associated with the various RL parameters based on final reward value).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the device of Simpson modified by Viegas by incorporating the reward threshold value parameter of Wang. Both concern the art reinforcement learning in traffic or congestion control, and the incorporation would have, according to Wang, allowing a termination condition based on reward value (p.5 ¶3, p.7 2nd ¶ from bottom), hence, better improving network performance (abstract).
Response to Arguments
Applicant’s arguments have been fully considered. In the remarks, Applicant argues:
1. The art of record, and Simpson in disclosure in disclosing tile coding, does not disclose the newly amended limitations directed to storing action values for discrete states.
Applicant’s arguments have been fully considered but are moot in view of the newly cited art.
2. Simpson teaches away from storing on discrete states.
While Simpson contemplates tile coding over a more straightforward implementation, such as Q-learning, for particular applications, instantiating additional applications, such as the channel allocation scheme to improve throughput on Wireless Mesh Networks scheme of Araqi, i.e., those having less states, would be motivated to implement the simpler RL techniques of Araqi.
3. Viegas does not remedy the shortcomings of Simpson.
Applicant’s arguments have been fully considered but are moot in view of the newly cited art.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Neystadt (US 20200128038 A1) discloses the use of RL including Q-learning (0139) in network security applications.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
/LIANG LI/
Primary examiner AU 2143
Prosecution Insights