Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Claim Rejections – 35 U.S.C. 103
Applicant’s prior art arguments have been fully considered but they are not persuasive.
Applicant argues (pg. 8) that the PER of Lin and methods of the present invention are fundamentally different sampling strategies.
Examiner respectfully disagrees. Applicant made a similar argument in the previous correspondence from the Applicant, and Examiner responded by stating that the difference between does not change the mechanics of the sampling algorithm itself. Applicant now responds by stating that the current amendments clarify this issue, but Examiner does not see which amendments exactly clarify this issue. Examiner believes that the same issues persist. Examiner suggests that Applicant points at the specific amendment that purportedly solves this issue and provide an argument as to why.
Applicant argues (pg. 8-9) that PER of Lin does not control data eviction by class, as the method of the present invention uses dedicated tables to protect rare data from being evicted.
Examiner respectfully disagrees. Applicant asserts that claims were amended to show how the data is protected from eviction based on the data for different scenarios being placed in different table partitions. However, Examiner does not see which amendments exactly clarify this issue. Examiner believes that the same issues persist. Examiner suggests that Applicant points at the specific amendment that purportedly solves this issue and provide an argument as to why.
Applicant argues (pg. 9) that PER of Lin is vulnerable to catastrophic forgetting in deep RL.
Examiner respectfully disagrees because this issue was already addressed in the previous office action with the addition of Korycki. See rejection below for more details (Korycki [Page 6, Col. 2, Paragraph 2]: “we can clearly see that RSB displayed stable incremental learning capabilities and was not affected by catastrophic forgetting”)
Applicant argues (pg. 9-10) that Song is different from the present invention because of the differences in the curriculum, data storage, sampling from the buffer, and event replays.
Examiner respectfully disagrees. Song primarily teaches “a reinforcement learning agent with mixed scenario” as well as “updating a policy for the AI agent to optimize the reward based upon data sampled from the experience replay buffer while the AI agent is simultaneously operating in the environment.” That the data is sampled from the experience replay buffer is taught by Lin, not Song. The motivation to combine is shown below in the rejection.
Applicant argues (pg. 10-11) that Korycki does not overcome the deficiencies of the aforementioned alleged deficiencies of claims 1, 14, and 20, as amended. Applicant states that while Korycki describes cluster driven replay buffers, these are different in form and functionality from the experience replay buffers as instantly described and claimed.
Examiner respectfully disagrees. In the previous office action, Examiner details that the experience replay buffer (RSB) of Korycki has clusters, which are analogous to tables. Examiner stated that the important feature is that distinct tables belong to correspondingly distinct classes, as distinct clusters belong to correspondingly distinct classes. This correspondence is maintained whether the information is organized into clusters or tables, and thus, the amended limitations with respect to Korycki is still taught by Korycki (see rejection below for updated explanation). Examiner suggests explaining why, particularly, the difference in form is also necessarily a difference in functionality between clusters and tables. Examiner views is as logically the same data and correspondence/relationships between data, just organized in a different form.
The foregoing applies to all independent claims and their dependent claims.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as "configured to" or "so that"; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step for”) in a claim with functional language creates a rebuttable presumption that the claim element is to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is invoked is rebutted when the function is recited with sufficient structure, material, or acts within the claim itself to entirely perform the recited function.
Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “trainer” in claim 14.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-14, 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US 20200238178 A1) hereinafter known as Lin in view of Song et al. (“Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning”) hereinafter known as Song in view of Korycki et al. (“Class-Incremental Experience Replay for Continual Learning under Concept Drift”) hereinafter known as Korycki.
Regarding independent claim 1, Lin teaches:
… providing an artificial intelligent (AI) agent in an environment having one or more predetermined scenario properties; (Lin [¶ 0042]: “Each playing environment is able to receive a player command (e.g., control command) from one or more player.” Lin teaches one or more players that are in a playing environment. These players are analogous to the AI agent in the environment. Lin [¶ 0042]: “Our experiment is trained on the unrealistic car racing game. This game have very complicate scene and road type which is different from the famous AI experimental game TORCS (The Open Racing Car Simulator)” Lin teaches that the environment is a car racing game with predetermined scenario properties, such as complicated scenes and road types.)
operating the AI agent in the environment while focusing on one or more specific skills; (Lin [¶ 0042]: “Each playing environment is able to receive a player command (e.g., control command) from one or more player.” Lin teaches one or more players that are in a playing environment. Lin [¶ 0109]: “The concept of our reward function is to punish all accidents that seriously decrease velocity, such as collisions, and encourage AI to drive faster.” Lin teaches that the players in the environment are operating by playing the game while focusing on skills, such as minimizing the accidents and maximizing velocity.)
providing a reward for successfully achieving the one or more specific skills; (Lin [¶ 0109]: “The concept of our reward function is to punish all accidents that seriously decrease velocity, such as collisions, and encourage AI to drive faster.” Lin teaches a reward function that factors in the specific skills, such as minimizing the accidents and maximizing velocity.)
…
…
…
Lin does not explicitly teach:
A method of training a reinforcement learning agent with mixed scenario training comprising: …
…
…
…
…
and updating a policy for the AI agent to optimize the reward based upon a batch of data sampled from the experience replay buffer, the batch constructed of data sampled from a proportion of data from each of the specific scenarios, while the AI agent is simultaneously operating in the environment.
However, Song teaches:
A method of training a reinforcement learning agent with mixed scenario training comprising: … (Song [Page 2, Column 2, Paragraph 2]: “High-speed overtaking in car racing involves two main objectives: minimizing the total overtaking time and avoiding collisions between the agent and other vehicles or obstacles” Song teaches a mixed scenario, namely one that involves two objectives, minimizing the total overtaking time and avoiding collisions with other vehicles, in the context of a racing game.)
…
…
…
…
and updating a policy for the AI agent to optimize the reward based upon a batch of data sampled from the experience replay buffer, the batch constructed of data sampled from a proportion of data from each of the specific scenarios, while the AI agent is simultaneously operating in the environment. (Song [Page 3, Column 1, Paragraph 2]: “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle” Song teaches that the policy is updated by maximizing the overtaking reward. This optimization function is described by a function that encourage the agent to “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle”. Song [Page 3, Column 2, Paragraph 2]: “In stage one, we train a policy (with random weights) for high-speed racing. … In stage two, we continuously train the same policy for aggressive racing and overtaking.” Song teaches creating a policy and training the policy so that is optimizes the reward of maximizing the velocity by racing and overtaking. Song [Page 3, Column 2, Paragraph 2]: “We update the policy by maximizing the overtaking reward (Eq. 2) and using new sampled trajectories” Since the update of the policy is being done by maximizing the overtaking function, which involves the AI agent to actively make approaches in the game, the updating of the policy is done while the AI agent is simultaneously operating in the game to generate new sampled trajectories.)
Lin and Song are in the same field of endeavor as the present invention, as the
references are directed to using reinforcement learning to train agents to optimally complete a racing car game. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine providing a reward function and a policy for the agent to optimize the way it races as taught in Lin with operating in an environment with multiple objectives and with multiple possible agents to train as taught in Song. Song provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Lin to include teachings of Song because the combination would allow for the reward function to include more than one objective and to train multiple agents. This has the potential benefit of being able to train agents in more complicated racing games which may emphasize multiple skills, such as speed and avoiding collisions.
Lin and Song do not explicitly teach:
streaming data from the AI agent to an experience replay buffer, wherein the data in the experience replay buffer is partitioned into one or more tables, wherein the experience replay buffer links specific scenarios to specific ones of the one or more tables and data related to each of the specific scenarios is stored in the linked specific one of the one or more tables;
re-weighting data in the experience replay buffer based on a pre-specified set of table weights to specify a proportion of a non-zero number of samples from each of the one or more tables to ensure data from shorter or rare scenarios is not ignored;
However, Korycki teaches:
streaming data from the AI agent to an experience replay buffer, wherein the data in the experience replay buffer is partitioned into one or more tables, wherein the experience replay buffer links specific scenarios to specific ones of the one or more tables and data related to each of the specific scenarios is stored in the linked specific one of the one or more tables; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches an experience replay buffer RSB that has clusters of classes. These clusters are analogous to tables, as the important feature is that distinct tables belong to correspondingly distinct classes, as distinct clusters belong to correspondingly distinct classes.)
re-weighting data in the experience replay buffer based on a pre-specified set of table weights to specify a proportion of a non-zero number of samples from each of the one or more tables to ensure data from shorter or rare scenarios is not ignored; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches RSB which ensures that there are at least cmin centroids for a class. This is a pre-specified minimum of centroids that need to be in a class. If the minimum is greater than 0, then this guarantees that there is a non-zero number of samples in that class. The updates are analogous to the reweighting of the data. Korycki [Page 6, Col. 2, Paragraph 2]: “we can clearly see that RSB displayed stable incremental learning capabilities and was not affected by catastrophic forgetting.” Korycki teaches that the effect of the replay buffer is to make sure that catastrophic forgetting, which is forgetting old or rare data, does not happen.)
Korycki is in the same field as the present invention, since it is directed to the algorithms of experience replay buffers to handle classes of scenarios. It would have been obvious, before the effective filing date of the claimed invention, to a person of ordinary skill in the art, to combine the ability for the reward function to include more than one objective and to train multiple agents as taught in Lin as modified by Song with separating the experience buffer by class as taught in Korycki. Korycki provides this additional functionality. As such, it would have been obvious to one of ordinary skill in the art to modify the teachings of Lin as modified by Song to include teachings of Korycki because the combination would allow for each class to be handled without a separate experience replay buffer and can all be done in one buffer. This has the potential benefit of saving memory in the operation, which can free up memory for other processes, thereby speeding up the overall processing.
Regarding dependent claim 4, Lin and Song teach:
The method of claim 1,
Song teaches:
wherein the scenario properties include one or more of launch conditions, opponent distribution options, a replication number, stopping conditions, experience table mapping and scenario weighting. (Song [Page 4, Column 2, Paragraph 4]: “We train three different agents using the overtaking reward with different hyperparameters and different training procedures for benchmark comparisons” Song teaches a scenario property where the replication number is specified – in particular, that the environment is populated by three different agents.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 5, Lin and Song teach:
The method of claim 1,
Song teaches:
further comprising launching an additional AI agent in an additional environment having a predetermined set of scenario properties. (Song [Page 3, Column 2, Paragraph 2]: “We use a randomly initialized neural network and train it for the single-player racing (without overtaking) … In stage two, we continuously train the same policy for aggressive racing and overtaking. We load the pre-trained policy done in the first stage and reconfigure the racing environment by adding an extra vehicle, which is controlled by the built-in game AI controller. We initialize the distance between our agent and the build-in agent with 200 meters separation along the centerline.” Song teaches that an additional AI agent is launched in stage two, when the overtaking is trained for. As this requires more than one player, an additional environment is essentially created by reconfiguring the racing environment, along with its scenario properties such as the separation from the centerline.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 6, Lin and Song teach:
The method of claim 5,
Song teaches:
wherein the predetermined set of scenario properties is randomly selected. (Song [Page 3, Column 2, Paragraph 2]: “Before training, it is important to keep the old replay buffer, and reinitialize the weights of the exploration term in the stochastic policy in that the policy maintains sufficient explorations. We update the policy by maximizing the overtaking reward (Eq. 2) and using new sampled trajectories” Song teaches that in the launching of the additional rollout worker, the weights in the stochastic (random/probabilistic) policy are reinitialized. This shows that the scenario properties, which include the stochastic policy, have a random component.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 7, Lin and Song teach:
The method of claim 5,
Song teaches:
wherein the predetermined set of scenario properties is selected based on a scenario weighting. (Song [Page 3, Column 2, Paragraph 2]: “Before training, it is important to keep the old replay buffer, and reinitialize the weights of the exploration term in the stochastic policy in that the policy maintains sufficient explorations. We update the policy by maximizing the overtaking reward (Eq. 2) and using new sampled trajectories” Song teaches that in the launching of the additional rollout worker, the weights in the stochastic (random/probabilistic) policy are reinitialized. The weights of the stochastic policy shows that the properties of the scenario are based on a weighting.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 8, Lin and Song teach:
The method of claim 5,
Song teaches:
wherein the predetermined set of scenario properties is automatically created from an event encountered from a prior AI agent in a prior environment. (Song [Page 3, Column 2, Paragraph 2]: “We use a randomly initialized neural network and train it for the single-player racing (without overtaking) … In stage two, we continuously train the same policy for aggressive racing and overtaking. We load the pre-trained policy done in the first stage and reconfigure the racing environment by adding an extra vehicle, which is controlled by the built-in game AI controller. We initialize the distance between our agent and the build-in agent with 200 meters separation along the centerline.” Song teaches that an additional player is created. When this is done, a predetermined subset of scenario properties, which include the 200 meter separation, is automatically created from the prior racing environment.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 9, Lin and Song teach:
The method of claim 1,
Song teaches:
further comprising providing a scenario properties of an opponent distribution that is well-behaved in the environment. (Song [Page 3, Column 2, Paragraph 2]: “We use a randomly initialized neural network and train it for the single-player racing (without overtaking) … In stage two, we continuously train the same policy for aggressive racing and overtaking. We load the pre-trained policy done in the first stage and reconfigure the racing environment by adding an extra vehicle, which is controlled by the built-in game AI controller. We initialize the distance between our agent and the build-in agent with 200 meters separation along the centerline.” Song teaches that an additional player is created, which is an opponent (because the original player is trying to overtake it.) The property of the opponent includes an initialization of the distance separating our agent and the build-in agent, which may be described as well-behaved.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 10, Lin and Song teach:
The method of claim 1,
Song teaches:
wherein the scenario properties include a replication number defining a number of parallel AI agents to operate in the environment. (Song [Page 4, Column 2, Paragraph 4]: “We train three different agents using the overtaking reward with different hyperparameters and different training procedures for benchmark comparisons.” Song teaches that a number of parallel players are operating in the environment. In doing so, Song specifies that the number of parallel players is a particular number, namely three, which is the replication number that defines this count.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 11, Lin and Song teach:
The method of claim 1,
Lin teaches:
wherein the scenario properties include a stopping condition. (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement.)
The reasons to combine are substantially siilar to those of claim 1.
Regarding dependent claim 12, Lin and Song teach:
The method of claim 11,
Lin teaches:
wherein the stopping condition is determined to generate an environment focused on a specific skill achievement. (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement. This puts the focus on maintaining speed as the primary skill of achievement.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 13, Lin and Song teach:
The method of claim 11,
Lin teaches:
wherein the stopping condition is open-ended, focusing the AI agent on achieving general techniques. (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement. This condition is open ended because it allows for the AI agent to achieve many different general techniques, with the only constraint being that speed cannot be too low for a long time.)
The reasons to combine are substantially similar to those of claim 1.
Regarding independent claim 14, Lin teaches:
… a set of artificial intelligent (AI) agents; (Lin [¶ 0042]: “Each playing environment is able to receive a player command (e.g., control command) from one or more player.” Lin teaches one or more players that are in a playing environment. These players are analogous to the rollout workers in the environment.)
a trainer; (Lin [¶ 0109]: “The concept of our reward function is to punish all accidents that seriously decrease velocity, such as collisions, and encourage AI to drive faster. We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches an element that enforces the reward function and the stopping condition, which can be considered as the trainer. This is because the trainer is the element that refines the policies of the rollout workers.)
and a set of scenario properties, wherein the trainer refines models and policies used to determine actions of each of the AI agents in an environment; (Lin [¶ 0109]: “The concept of our reward function is to punish all accidents that seriously decrease velocity, such as collisions, and encourage AI to drive faster. We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches an element that enforces the reward function and the stopping condition, which can be considered as the trainer. This is because the trainer is the element that refines the policies of the rollout workers.)
each of the AI agents operating in the environment based on predetermined launch conditions retrieved from the scenario properties; (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement. This is a launch condition, as the stopping condition is determined at the start of the race and determines the course of the race.)
Song teaches:
A deep reinforcement learning architecture using mixed scenario training comprising: … (Song [Page 2, Column 2, Paragraph 2]: “High-speed overtaking in car racing involves two main objectives: minimizing the total overtaking time and avoiding collisions between the agent and other vehicles or obstacles” Song teaches a mixed scenario, namely one that involves two objectives, minimizing the total overtaking time and avoiding collisions with other vehicles, in the context of a racing game.)
…
…
…
…
…
and performing a policy refinement for the AI agent to optimize the reward based upon data sampled from the experience replay buffer while the AI agent is simultaneously operating in the environment. (Song [Page 3, Column 1, Paragraph 2]: “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle” Song teaches that the policy is updated by maximizing the overtaking reward. This optimization function is described by a function that encourage the agent to “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle”. Song [Page 3, Column 2, Paragraph 2]: “In stage one, we train a policy (with random weights) for high-speed racing. … In stage two, we continuously train the same policy for aggressive racing and overtaking.” Song teaches creating a policy and training the policy so that is optimizes the reward of maximizing the velocity by racing and overtaking. Song [Page 3, Column 2, Paragraph 2]: “We update the policy by maximizing the overtaking reward (Eq. 2) and using new sampled trajectories” Since the update of the policy is being done by maximizing the overtaking function, which involves the AI agent to actively make approaches in the game, the updating of the policy is done while the AI agent is simultaneously operating in the game to generate new sampled trajectories.)
Korycki teaches:
data from each of the AI agents operating in the environment with the predetermined launch conditions is collected and stored in an experience replay buffer of the trainer, wherein the data in the experience replay buffer is partitioned into one or more tables, wherein the experience replay buffer links specific scenarios to specific ones of the one or more tables; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches an experience replay buffer RSB that has clusters of classes. These clusters are analogous to tables, as the important feature is that distinct tables belong to correspondingly distinct classes, as distinct clusters belong to correspondingly distinct classes.)
re-weighting data in the experience replay buffer based on a pre-specified set of table weights to specify a proportion of data from each of the one or more tables; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches RSB which ensures that there are at least cmin centroids for a class. This is a pre-specified minimum of centroids that need to be in a class. The updates are analogous to the reweighting of the data.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 16, Lin and Song teach:
The deep reinforcement learning architecture of claim 15,
Lin teaches:
wherein the experience replay buffer includes tables for partitioning the data. (Lin [¶ 0096]: “For an off-policy learning algorithm, an experience replay is commonly used to store experiences for future training. An experience consists of a 4-tuple (st, at, rt, st+1), including a state st, an action at, a reward rt at time t, a next state st+1 at time (t+1).” Lin teaches taking the experiences of the players at a state and storing the experiences in an experience replay. Lin teaches that each experience is a tensor, which shows that the experience replay must partitioned into a table that contains tensors, in order to store the experiences.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 17, Lin and Song teach:
The deep reinforcement learning architecture of claim 16,
Lin teaches:
wherein the batch of data includes data from multiple ones of the tables, wherein each table is provided a predetermined table weight. (Lin [¶ 0106]: “experiences with high prediction error may contribute more knowledge for learning, so the sampling probability is proportional to the prediction error” Lin teaches that the data, that may be in table form, each have a sampling probability. This is essentially a weight of the table that comprises the data, as the likelihood of choosing that table is proportional to the sampling probability.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 18, Lin and Song teach:
The deep reinforcement learning architecture of claim 14,
Lin teaches:
wherein the trainer includes a task manager module for determine which scenario properties should be used by an idle one of the set of AI agents. (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement – clearly this is used by an idle AI agent.)
The reasons to combine are substantially similar to those of claim 1.
Regarding dependent claim 19, Lin and Song teach:
The deep reinforcement learning architecture of claim 14,
Lin teaches:
wherein the data in the experience replay buffer includes a state, an action and rewards of each of the AI agents. (Lin [¶ 0096]: “For an off-policy learning algorithm, an experience replay is commonly used to store experiences for future training. An experience consists of a 4-tuple (st, at, rt, st+1), including a state st, an action at, a reward rt at time t, a next state st+1 at time (t+1).” Lin teaches taking the experiences of the players at a state and storing the experiences in an experience replay. Lin teaches that each experience has a state, action, and rewards.)
The reasons to combine are substantially similar to those of claim 1.
Regarding independent claim 20, Lin teaches:
…
…
and training the agent in an environment with predefined scenario properties, wherein the predefined scenario properties includes launch conditions, … stopping conditions, experience table mapping and scenario weighting, and wherein the training includes: (Lin [¶ 0109]: “We also use early stopping method that we terminate the episode immediately when the velocity remains low after several actions done.” Lin teaches a stopping condition, where the episode terminates if the velocity remains low for multiple actions. This is put into place to discourage long periods of time with little to no movement. This is also a launch condition, as the stopping condition is determined at the start of the race and determines the course of the race. Lin [¶ 0106]: “experiences with high prediction error may contribute more knowledge for learning, so the sampling probability is proportional to the prediction error” Lin teaches that the data, that may be in table form, each have a sampling probability. This is essentially a weight of the table that comprises the data, as the likelihood of choosing that table is proportional to the sampling probability. Thus, this is the experience table mapping and the scenario weighting.)
Song teaches:
A method of training an agent with deep reinforcement learning to interact in a racing video game, comprising: learning a policy that selects an action based on observations by the agent and based on a value function that estimates a future rewards for each possible action; (Song [Page 3, Column 2, Paragraph 2]: “In stage one, we train a policy (with random weights) for high-speed racing. … In stage two, we continuously train the same policy for aggressive racing and overtaking … In stage three, we obtain a final policy that can race the car at high speed, overtake its opponents, and avoid collisions.” Song teaches creating a policy and training the policy so that is optimizes the reward of maximizing the velocity by racing and overtaking. Song teaches that the policy function can avoid collisions, which show that the future rewards can be calculated.)
mapping core actions of the agent to either a changing velocity dimension and a steering dimension, wherein the changing velocity dimension and the steering dimension are both continuous-valued dimensions; (Song [Page 3, Column 1, Paragraph 3]: “We denote the observation vector … and use them as the input to our neural networks. The control actions are the steering angle and the combined throttle and brake signal, denoted as [δt, ωt] separately” Song teaches that the observation vector comprises the steering angle. This is used as the input to the neural network which trains the actions of the agent.)
… opponent distribution options, a replication number, … (Song [Page 4, Column 2, Paragraph 4]: “We train three different agents using the overtaking reward with different hyperparameters and different training procedures for benchmark comparisons.” Song teaches that a number of parallel players are operating in the environment. In doing so, Song specifies that the number of parallel players is a particular number, namely three, which is the replication number that defines this count. This also is an option that determines the distribution of the opponents.)
…
…
and updating a policy for the AI agent to optimize the reward based upon data sampled from the experience replay buffer while the AI agent is simultaneously operating in the environment. (Song [Page 3, Column 1, Paragraph 2]: “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle” Song teaches that the policy is updated by maximizing the overtaking reward. This optimization function is described by a function that encourage the agent to “approach the front opponent vehicle (i) when it is driving behind, while keep maximizing the relative distance once it has overtaken the opponent vehicle”. Song [Page 3, Column 2, Paragraph 2]: “In stage one, we train a policy (with random weights) for high-speed racing. … In stage two, we continuously train the same policy for aggressive racing and overtaking.” Song teaches creating a policy and training the policy so that is optimizes the reward of maximizing the velocity by racing and overtaking. Song [Page 3, Column 2, Paragraph 2]: “We update the policy by maximizing the overtaking reward (Eq. 2) and using new sampled trajectories” Since the update of the policy is being done by maximizing the overtaking function, which involves the AI agent to actively make approaches in the game, the updating of the policy is done while the AI agent is simultaneously operating in the game to generate new sampled trajectories.)
Korychi teaches:
streaming data from the AI agent to an experience replay buffer, wherein the data in the experience replay buffer is partitioned into one or more tables, wherein the experience replay buffer links specific scenarios to specific ones of the one or more tables; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches an experience replay buffer RSB that has clusters of classes. These clusters are analogous to tables, as the important feature is that distinct tables belong to correspondingly distinct classes, as distinct clusters belong to correspondingly distinct classes.)
re-weighting data in the experience replay buffer based on a pre-specified set of table weights to specify a proportion of data from each of the one or more tables; (Korycki [Page 4, Column 1, Paragraph 4]: “we propose a modification of the clustering-driven replay buffers, called Re active Subspace Buffer (RSB) … In the given algorithm, for each new instance x with a label y, we first ensure that there are at least cmin centroids for the class. Then, we find the nearest cluster Cx for the given instance x. If the given cluster belongs to the class y of the instance, we simply update it, its buffer Bx of maximum size bmax and sliding window Wx of maximum size ωmax, where the last component is responsible for tracking the most current concepts for the given centroid. Otherwise, there is a risk that a concept drift appeared and instances of a different class have started appearing around the centroid. Therefore, if the instance x is sufficiently close (we use simple standard deviation rules), we update the sliding window of the centroid Cx, but not the cluster itself.” Korycki teaches RSB which ensures that there are at least cmin centroids for a class. This is a pre-specified minimum of centroids that need to be in a class. The updates are analogous to the reweighting of the data.)
The reasons to combine are substantially similar to those of claim 1.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYU HYUNG HAN whose telephone number is (703) 756-5529. The examiner can normally be reached on MF 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Kyu Hyung Han/
Examiner
Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights