DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-19 are presented for examination based on the application filed on April 6, 2023.
Claims 10-15 and 17 are rejected under 35 U.S.C. § 112(b) or 35 U.S.C. § 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. § 112, the applicant), regards as the invention.
Claims 1-19 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to judicial exception, an abstract idea, and it has not been integrated into practical application. The claims further do not recite significantly more than the judicial exception.
Claims 1-7, 10-12, and 14-19 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Sharma, Anil, Mayank K. Pal, Saket Anand, and Sanjit K. Kaul. "Stratified sampling based experience replay for efficient camera selection decisions." In 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), pp. 144-151. IEEE, 2020 [herein “Sharma”].
Claim 8 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Wurman, Peter R., Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco et al. "Outracing champion Gran Turismo drivers with deep reinforcement learning." Nature 602, no. 7896 (2022): 223-228.
Claim 9 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Luo, Jieliang, and Hui Li. "Dynamic experience replay." In Conference on robot learning, pp. 1191-1200. PMLR, 2020 [herein “Luo”].
Claim 13 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Daley, Brett, Cameron Hickert, and Christopher Amato. "Stratified experience replay: Correcting multiplicity bias in off-policy reinforcement learning." arXiv preprint arXiv:2102.11319 (2021) [herein “Daley”].
This action is made non-Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on April 6, 2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because of the following:
FIG. 1 since the legend has two iterations for “Event Table Data”. One of these should be default table data (see Para. 0034). Also, the legend does not provide a means to differential the two iterations of “Event Table Data” in the drawing as both iterations are represented by two hollow squares (e.g., “◻+◻”).
FIG. 1, FIG. 2A-2E, FIG. 3A, FIG. 4A, FIG. 6A-6E, FIG. 7A-7E, FIG. 8A, FIG. 9-10, FIG. 11A-11C, FIG. 13A-13D, and FIG. 17A fail to show “blue squares”, “blue lines”, “green square”, “shaded region”, “yellow squares”, “orange squares” and “red lines” as described in the specification. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing (see MPEP § 608.02(d) and 37 CFR 1.83(a)).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of a n amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin a s either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
Para. 0081, which cites “FIFO”, should be “first in first out (FIFO)”.
Para. 0084, which cites “MDPs”, should be “Markov Decision Processes (MDPs)”.
Para. 0099, which cites “increasing the number of event conditioned tables n”, should be “increasing the number of n event conditioned tables”.
Para. 00113, which cites “GP”, should be “Grand Prix (GP)”.
Para. 00113, which cites “GT”, should be “Gran Turismo Sport®”.
The use of the terms “DVD”, “Bluetooth”, “OpenAi”, “Red Bull”, and “Porsche”, which are trade names or marks used in commerce, have been noted in this application. The terms should be accompanied by the generic terminology; furthermore, the terms should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term. Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Appropriate correction is required.
Claim Objections
Claims 1-10, 12-15, and 17 are objected to because of the following informality: recitations of elements with no previous recitations. For example, claim 1, “the given history length” in Ln. 8, is improper because there has been no previous recitation of “the given history length”. For the purpose of examination, “the given history length” will be interpreted as “the ”. Similarly, the following are objected under similar rationale:
Claim 1, “each table” in Ln. 7, should be “each event table”, respectively. Claims 2-21 are also objected to for incorporating the deficiency of its independent claim 1.
Claim 2, “The method” in Ln. 10, should be “The computer-implemented method”. Claims 3-10 and 12-15, having similar limitations of claim 2, are also objected.
Claim 17, “The computer-implemented method” in Ln. 15, should be “The ”.
All claims dependent on an objected base claim are objected based on their dependency
Appropriate correction is required.
Claim Rejections - 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. § 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10-15 and 17-18 are rejected under 35 U.S.C. § 112(b) or 35 U.S.C. § 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. § 112, the applicant), regards as the invention.
Claim 10 recites the term “suboptimal”, which is a relative term that renders the claim indefinite. The term “suboptimal” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention (See MPEP § 2173.05(b)).
Claim 11 recites “for utilizing the event tables with a reinforcement learning system” in Ln 12-13. This phrase renders the claim indefinite, because it merely recites a use without any steps delimiting the use (See MPEP § 2173.05(q), “Attempts to claim a process without setting forth any steps involved in the process generally raises an issue of indefiniteness under 35 U.S.C. § 112(b) or pre-AIA 35 U.S.C. § 112, second paragraph”). Claim 17, having similar limitations of claim 11, is also rejected under the similar rationale. Claims 12-15 and claim 18, being dependent on claims 11 and 17, respectively, are also rejected.
Claim 12 recites the term “sufficient”, which is a relative term that renders the claim indefinite. The term “sufficient” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention (See MPEP § 2173.05(b)).
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to judicial exception, an abstract idea, and it has not been integrated into practical application. The claims further do not recite significantly more than the judicial exception. Examiner has evaluated the claims under the framework provided in the 2019 Patent Eligibility Guidance published in the Federal Register 01/07/2019 and has provided such analysis below.
Step 1:
Claims 1-10 are directed to a method and fall within the statutory category of a process; claims 11-15 are directed to a method and fall within the statutory category of a process; and claims 16-19 are directed to a method and fall within the statutory category of a process. Therefore, “Are the claims to a process, machine, manufacture or composition of matter?” Yes.
In order to evaluate the Step 2A inquiry “Is the claim directed to a law of nature, a natural phenomenon or an abstract idea?” we must determine, at Step 2A Prong 1, whether the claim recites a law of nature, a natural phenomenon or an abstract idea and further whether the claim recites additional elements that integrate the judicial exception into a practical application.
Step 2A Prong 1:
Claims 1, 11, and 16: The limitations of:
“determining at least one event condition”,
“determining at least one preceding state indicating a state preceding the event condition within a history length”,
“partitioning an experience replay buffer for a reinforcement learning agent into event tables based on event conditions and history lengths, wherein each table contains data where a corresponding event condition was true or states that preceded the corresponding event condition within the given history length”, and
“permitting stratified sampling from the event tables (SSET) for utilizing the event tables with a reinforcement learning system”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, the limitations can be conducted as the following:
a person can mentally determine or draw with a pencil and paper from a data table a key event such as a racecar winning a race,
a person can mentally determine or draw with a pencil and paper from a data table containing a history of events and actions such as racecar placement prior to winning a race,
a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables that include key events and actions that proceeded the key event,
a person can mentally sample or draw with a pencil and paper the data from the smaller tables to provide a fair representation of the events that occur for use in training a RL.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Therefore, yes, claims 1, 11, and 16 recite judicial exceptions. The claims have been identified to recite judicial exceptions, Step 2A Prong 2 will evaluate whether the claims are directed to the judicial exception.
Step 2A Prong 2:
Claims 1, 11, and 16: The judicial exception is not integrated into a practical application. In particular, the claims recite the following additional elements: “A computer-implemented method” which is merely a recitation of generic computing components and functions being used as a tool to implement the judicial exception (see MPEP § 2106.05(f)) with the broadest reasonable interpretation, which does not integrate a judicial exception into elements.
Therefore, “Do the claims recite additional elements that integrate the judicial exception into a practical application?” No, these additional elements do not integrate the abstract idea into a practical application and they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
After having evaluated the inquires set forth in Steps 2A Prong 1 and 2, it has been concluded that claims 1, 11, and 16 not only recite a judicial exception but that the claims are directed to the judicial exception as the judicial exception has not been integrated into practical application.
Step 2B:
Claims 1, 11, and 16: The claims do not include additional elements, alone or in combination, that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than generic computing components.
Therefore, “Do the claims recite additional elements that amount to significantly more than the judicial exception?” No, these additional elements, alone or in combination, do not amount to significantly more than the judicial exception. Having concluded the analysis within the provided framework, claims 1, 11, and 16 do not recite patent eligible subject matter under 35 U.S.C. § 101.
Regarding claims 2 and 19, they recite an additional limitations of “where the experience replay buffer includes a default table that holds all incoming data”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table containing all data for a race into separate smaller tables that include key events and actions that proceeded the key event.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 3, it recites an additional limitation of “where the event conditions are based on histories and not just a single state”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables that include key events and actions that proceeded the key event.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 4, it recites an additional limitation of “where each event table has a specified capacity in the experience replay buffer proportional to an overall size of the experience replay buffer”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables of equal size that include key events and actions that proceeded the key event.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 5, it recites an additional limitation of “where each table has a specified sampling probability”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables that include key events and actions that proceeded the key event based on how often the key events occurred in the race.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 6, it recites an additional limitation of “wherein a mapping from the event conditions to the event tables is surjective, with multiple ones of the event conditions funneling data into the same one of the event tables”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables having a certain type of key events and actions, such as a crashes and passing, and where each key event and action are only placed into one.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 7, it recites an additional limitation of “where the event conditions represent a handling or manipulation of objects by an artificial agent”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables that include key events and actions that proceeded the key event, where the key event was previously specified by an artificial agent.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 8, it recites an additional limitation of “wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream, winning a race, recovering from going off-course, and racing incidents”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally determine or draw with a pencil and paper from a data table containing data from a car racing simulator a key event such as a racecar winning a race, a racecar drafting in a car’s slipstream, recovering from going off-course, and crashes.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 9, it recites an additional limitation of “wherein, in continuous control problems that already have rich reward signals, event conditions are based on exceeding thresholds in immediate rewards from a state”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally determine or draw with a pencil and paper from a data table containing data an event such as a racecar winning a race whose reward from reinforcement learning was higher than a certain threshold to find key events.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 10, it recites an additional limitation of “wherein at least some of the event conditions are suboptimal and designed to help the artificial agent retain memory of, and avoid, adverse outcomes”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally separate or draw with a pencil and paper a data table into separate smaller tables that include key events and actions that proceeded the key event, where at least some key events whose reward from reinforcement learning was lower than a certain threshold to key events higher than the threshold be of more importance.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claims 12 and 18, they recite an additional limitation of “blocking sampling from one of the event tables until it has sufficient data”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally sample or draw with a pencil and paper the data from the smaller tables to provide a fair representation of the events that occur for use in training a RL only after the smaller tables have been completely generated.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 13, it recites an additional limitation of “determining a bias correction term for stochastic environments”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally sample or draw with a pencil and paper the data from the smaller tables to provide a fair representation of the events by using and generating a term to balance any bias from a stochastic system.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 14, it recites an additional limitation of “applying a prioritization scheme inside each of the event tables while sampling”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally sample or draw with a pencil and paper the data from the smaller tables to provide a fair representation of the events based on how often the key events occurred in the race.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 15, it recites an additional limitations of “multi-task training to balance gradient updates to respect data from each of the event tables” as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation of mathematical evaluations. For example, calculating gradients to train a machine learning can be accomplished using a Conflict-Averse Gradient descent (Para. 0077-0079 and 102-00103 have the equations for determining and using gradient updates from Gradient descent). Furthermore, this current claim is analogous to Example 47 for training a machine learning model using a gradient descent algorithm. Similarly in Example 47, the claim limitation was found to be a mathematical concept (see July 2024 Subject Matter Eligibility Examples, https://www.uspto.gov/sites/default/files/documents/2024-AI-SMEUpdateExamples47-49.pdf).
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation of mathematic evaluations but for the recitation of generic computer components, then it falls within the “Mathematical Operation” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Regarding claim 17, it recites an additional limitation of “permitting stratified sampling from the event tables (SSET) for utilizing the event tables with a reinforcement learning system”, as drafted, is a process that, but for the recitation of generic computing components, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper. For example, a person can mentally sample or draw with a pencil and paper the data from the smaller tables to provide a fair representation of the events based on how often the key events occurred in the race.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind or with pen and paper but for the recitation of generic computer components, then it falls within the “Mental Processes” grouping of abstract ideas. Accordingly, the claim recites an abstract idea under Prong I step 2A.
Therefore, having concluded the analysis within the provided framework, claims 1-19 do not recite patent eligible subject matter and are rejected under 35 U.S.C. § 101 because the claimed invention is directed to judicial exception, an abstract idea, that has not been integrated into a practical application. The claims further do not recite significantly more than the judicial exception. Claims 2-10, 12-15, and 17-19 are also rejected for incorporating the deficiency of their dependent claims 1, 11, and 16.
Claim Rejections - 35 U.S.C. § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. § 102 and 103 (or as subject to pre-AIA 35 U.S.C. § 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7, 10-12, and 14-19 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Sharma, Anil, Mayank K. Pal, Saket Anand, and Sanjit K. Kaul. "Stratified sampling based experience replay for efficient camera selection decisions." In 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), pp. 144-151. IEEE, 2020 [herein “Sharma”].
As per claim 1, Sharma teaches “A computer-implemented method comprising: partitioning an experience replay buffer for a reinforcement learning agent into event tables based on event conditions and history lengths, wherein each table contains data where a corresponding event condition was true or states that preceded the corresponding event condition within the given history length.” (Pg. 145 Sect. I, “This is an important observation with respect to training a deep RL model, which requires appropriately handling of imbalanced state transitions during experience replay” [A method comprising an experience replay buffer for a reinforcement learning agent]. “Our proposed approach, referred to as Stratified Experience Replay (SER) resolves this challenge by sampling in the imbalanced replay memory created by the different episodic runs of the agent-environment interaction. Our specific contributions are the following: 1) We propose a novel experience replay method to segregate transitions into multiple replay memories” [partitioning an experience replay buffer for a reinforcement learning agent into event tables]. Pg. 146 Sect. III, “during transitions or occlusions, the state at time t captures three elements to handle this partially observable state of the target: 1) xt: the last observed location of the target…, 2) ht: the action history that maintains a list of previously selected actions by the policy. The history is stored as a list of cameras encoded as a one-hot vector. 3) τ: This is the time-elapsed vector that captures the time since the target’s most recent observation by the agent in any camera” [based on event conditions and history lengths]. Furthermore, Pg. 147 Sect. IV, “Therefore, we have segregated transitions into multiple replay memories to enable sampling of all kinds of transitions to create the minibatch. Also as observed in supervised learning, we need present the rare transitions more often to the network. To ensure efficient sampling of rare and other transitions, we segregate transitions in different replay memories to compensate for searching in significantly large replay memory” [partitioning an experience replay buffer for a reinforcement learning agent into event tables based on event conditions and history lengths]. Pg. 148 Sect. IV, “The multiple replays are; Rf, which stores the transitions which pertain frequently occurring action C×; R−, which stores the transitions that receive a negative reward, and R+ which stores the transitions that receive a positive reward” [e.g., wherein each table contains data where a corresponding event condition was true]. Pg. 148 Sect. V, “We implemented the DQN algorithm using PyTorch framework and utilized a server with 128 GBs of RAM and a 11-GB Nvidia RTX 2080 Ti GPU for training” [A computer-implemented method]. Further see Sect. I and III-V. The examiner has interpreted that using PyTorch framework and a server for training a deep reinforcement learning (RL) model to handle state transition during experience replay by segregating transitions into multiple relay memories to create a minibatch to compensate for searching in large replay memory where transitions have states that capture locations of targets, previous selected actions, time of last captured location, and store transitions which pertain frequently occurring action as a computer-implemented method comprising: partitioning an experience replay buffer for a reinforcement learning agent into event tables based on event conditions and history lengths, wherein each table contains data where a corresponding event condition was true.)
As per claim 2, Sharma teaches “where the experience replay buffer includes a default table that holds all incoming data.” (Pg. 147 Sect. IV, “replay memory of size R is used, which stores the last |R| transitions” [where the experience replay buffer includes a default table that holds all incoming data]. Further see Sect. IV.)
As per claim 3, Sharma teaches “where the event conditions are based on histories and not just a single state.” (Pg. 146 Sect. III, “during transitions or occlusions, the state at time t captures three elements to handle this partially observable state of the target: 1) xt: the last observed location of the target…, 2) ht: the action history that maintains a list of previously selected actions by the policy. The history is stored as a list of cameras encoded as a one-hot vector. 3) τ: This is the time-elapsed vector that captures the time since the target’s most recent observation by the agent in any camera” [where the event conditions are based on histories and not just a single state]. Further see Sect. III-V. The examiner has interpreted that transitions previous selected actions in an action history as where the event conditions are based on histories and not just a single state.)
As per claim 4, Sharma teaches “where each event table has a specified capacity in the experience replay buffer proportional to an overall size of the experience replay buffer.” (Pg. 147 Sect. IV, “Please note that state of-the-art replay methods use single replay memory with recommended size of 106” [an overall size of the experience replay buffer]. Pg. 148 Sect. V, “For the proposed method, we used three replays to separate the rare transitions, where the size of each replay was set to 103 acquired via hyperparameter tuning” [e.g., where each event table has a specified capacity in the experience replay buffer proportional to an overall size of the experience replay buffer]. Further see Sect. IV-V. The examiner has interpreted that separating a single replay memory with recommended size of 106 into three separate replays to store transitions of size 103 as where each event table has a specified capacity in the experience replay buffer proportional to an overall size of the experience replay buffer.)
As per claim 5, Sharma teaches “where each table has a specified sampling probability.” (Pg. 148 Sect. IV, “The multiple replays are; Rf, which stores the transitions which pertain frequently occurring action C×; R−, which stores the transitions that receive a negative reward, and R+ which stores the transitions that receive a positive reward” [e.g., wherein each table contains certain transitions]. Pg. 148 Sect. IV, “The transitions are sampled uniformly from Rf, and R− replay. R+ stores both rare and other positive reward transitions, it follows a prioritized sampling. For priority sampling, a higher weight is assigned to the rare transitions and a lower weight to other transitions. A probability value is assigned to ith transition as
w
i
/
∑
w
i
” [transitions are sampled having uniform probability and others have a probability value, e.g. where each table has a specified sampling probability]. Further see Sect. IV. The examiner has interpreted that having multiple replays having a uniform and a priority sampling with a probability have that are sampled into three replays as where each table has a specified sampling probability.)
As per claim 6, Sharma teaches “wherein a mapping from the event conditions to the event tables is surjective, with multiple ones of the event conditions funneling data into the same one of the event tables.” (Pg. 148 Sect. IV “The multiple replays are; Rf, which stores the transitions which pertain frequently occurring action C×; R−, which stores the transitions that receive a negative reward, and R+ which stores the transitions that receive a positive reward…This transition is stored in a replay which satisfies the above criteria for segregation” [wherein a mapping from the event conditions to the event tables is surjective]. The examiner would like to refer the applicant to Figure 3 from Sharma, shown below as Figure 1, that depicts that transitions are stored in one of three replay memories. As shown, if rt+1 = rcx, then the transition is stored in Rf. If rt+1 < rcx, then the transition is stored in R_. If rt+1 > rcx, then the transition is stored in R+. Additionally, this is also seen in Algorithm 1, shown below as Figure 2, annotated lines 15-21 shown below that shows that transitions are binned into one of three replay memories based on the reward received. This shows that transitions can get binned into the same replay memories, e.g., with multiple ones of the event conditions funneling data into the same one of the event tables. Further see Sect. IV. The examiner has interpreted that storing transitions into one of three replay memories based on the received rewards as wherein a mapping from the event conditions to the event tables is surjective, with multiple ones of the event conditions funneling data into the same one of the event tables.)
PNG
media_image1.png
508
724
media_image1.png
Greyscale
Figure 1: Figure 3 - Overview of the proposed experience replay method
[AltContent: textbox (transitions are stored in one of three replay memories)][AltContent: ][AltContent: textbox (1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35)]
PNG
media_image2.png
930
565
media_image2.png
Greyscale
Figure 2: Algorithm 1 DQN with Stratified Experience Replay
As per claim 7, Sharma teaches “where the event conditions represent a handling or manipulation of objects by an artificial agent.” (Pg. 146 Sect. IV, “We model camera selection decision as a finite horizon discounted sum reward problem, where an RL agent is responsible for deciding the presence of a target given a camera frame at a discrete time step t. At each time step, the agent receives the location of the target to be tracked contained in st. The agent needs to select one of the cameras represented using the action space A = {0,1,...,N,C×}. The agent interacts with the environment Ɛ by selecting an action” [where the event conditions represent a handling of objects by an artificial agent]. Further see Sect. IV.)
As per claim 10, Sharma teaches “wherein at least some of the event conditions are suboptimal and designed to help the artificial agent retain memory of, and avoid, adverse outcomes.” (Pg. 148 Sect. IV “The multiple replays are; Rf, which stores the transitions which pertain frequently occurring action C×; R−, which stores the transitions that receive a negative reward, and R+ which stores the transitions that receive a positive reward…This transition is stored in a replay which satisfies the above criteria for segregation” [e.g., wherein at least some of the event conditions are suboptimal and designed to help the artificial agent retain memory of, and avoid, adverse outcomes]. Further see Sect. IV.)
Re Claim 11, it is a method claim, having similar limitations of claim 1. Thus, claim 11 is also rejected under the similar rationale as cited in the rejection of claim 1.
Furthermore, regarding claim 11, Sharma teaches “permitting stratified sampling from the event tables (SSET) for utilizing the event tables with a reinforcement learning system.” (Pg. 145 Sect. I, “This is an important observation with respect to training a deep RL model, which requires appropriately handling of imbalanced state transitions during experience replay” [A method comprising an experience replay buffer for a reinforcement learning agent]. “Our proposed approach, referred to as Stratified Experience Replay (SER) resolves this challenge by sampling in the imbalanced replay memory created by the different episodic runs of the agent-environment interaction. Our specific contributions are the following: 1) We propose a novel experience replay method to segregate transitions into multiple replay memories. Our investigations show that stratified sampling helps learning a better policy for camera selection in a camera network” [permitting stratified sampling from the event tables (SSET) for utilizing the event tables with a reinforcement learning system]. Further see Sect. I. The examiner has interpreted that using stratified sampling to help train a deep reinforcement learning model using the segregate transitions into multiple memories as permitting stratified sampling from the event tables (SSET) for utilizing the event tables with a reinforcement learning system.)
As per claim 12, Sharma teaches “blocking sampling from one of the event tables until it has sufficient data.” (Pg. 148 Sect. V, “For the proposed method, we used three replays to separate the rare transitions, where the size of each replay was set to 103 acquired via hyperparameter tuning” [event tables size or amount of data]. Pg. 147 Sect. IV, “To ensure efficient sampling of rare and other transitions, we segregate transitions in different replay memories to compensate for searching in significantly large replay memory. We create three replay memories named Rf, R+, and R−. Given the three replay memories, we sample a minibatch” [create the tables first then sample, e.g., blocking sampling from one of the event tables until it has sufficient data]. Furthermore, Pg. 148 Sect. IV, “This transition is stored in a replay which satisfies the above criteria for segregation. For learning, a minibatch is prepared from the stored transitions in the multiple replays. The transitions are sampled uniformly from Rf and R− replay. R+ stores both rare and other positive reward transitions, it follows a prioritized sampling” [storing then sampling, e.g., blocking sampling from one of the event tables until it has sufficient data]. Further see Sect. IV-V. The examiner has interpreted that segregating transitions into three separate replays of size 103 and then sampling the transitions of the multiple replays as blocking sampling from one of the event tables until it has sufficient data.)
As per claim 14, Sharma teaches “applying a prioritization scheme inside each of the event tables while sampling.” (Pg. 148 Sect. IV, “The transitions are sampled uniformly from Rf, and R− replay. R+ stores both rare and other positive reward transitions, it follows a prioritized sampling. For priority sampling, a higher weight is assigned to the rare transitions and a lower weight to other transitions. A probability value is assigned to ith transition as
w
i
/
∑
w
i
” [applying a prioritization scheme inside each of the event tables while sampling]. Further see Sect. IV. The examiner has interpreted that having multiple replays having a uniform and a priority sampling with a probability have that are sampled into three replays as applying a prioritization scheme inside each of the event tables while sampling.)
As per claim 15, Sharma teaches “multi-task training to balance gradient updates to respect data from each of the event tables.” (As shown above Figure 2, line 32 performs gradient descent updates. Pg. 150 Sect. VI, “Our experiments showed that SER based DQN resulted in high recall even in camera networks that have long transition times, thus showing that SER successfully balanced rare and frequent actions while learning a policy for camera selection” [multi-task training to balance gradient updates to respect data from each of the event tables]. Further see Sect. IV and VI. The examiner has interpreted that performing gradient descent updates to successfully balance rare and frequent actions while learning a policy for camera selection as multi-task training to balance gradient updates to respect data from each of the event tables.)
Re Claim 16, it is a method claim, having similar limitations of claim 1. Thus, claim 16 is also rejected under the similar rationale as cited in the rejection of claim 1.
Furthermore, regarding claim 16, Sharma teaches “determining at least one event condition; determining at least one preceding state indicating a state preceding the event condition within a history length”. (Pg. 146 Sect. III, “during transitions or occlusions, the state at time t captures three elements to handle this partially observable state of the target: 1) xt: the last observed location of the target…, 2) ht: the action history that maintains a list of previously selected actions by the policy. The history is stored as a list of cameras encoded as a one-hot vector. 3) τ: This is the time-elapsed vector that captures the time since the target’s most recent observation by the agent in any camera” [determining at least one event condition; determining at least one preceding state indicating a state preceding the event condition within a history length]. Further see Sect. III. The examiner has interpreted that transitions that have states that capture locations of targets, previous selected actions, time of last captured location, and store transitions which pertain frequently occurring action as determining at least one event condition; determining at least one preceding state indicating a state preceding the event condition within a history length.)
Re Claim 17, it is a method claim, having similar limitations of claim 11. Thus, claim 17 is also rejected under the similar rationale as cited in the rejection of claim 11.
Re Claim 18, it is a method claim, having similar limitations of claim 12. Thus, claim 18 is also rejected under the similar rationale as cited in the rejection of claim 12.
Re Claim 19, it is a method claim, having similar limitations of claim 2. Thus, claim 19 is also rejected under the similar rationale as cited in the rejection of claim 2.
Claim Rejections - 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. § 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claim 8 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Wurman, Peter R., Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco et al. "Outracing champion Gran Turismo drivers with deep reinforcement learning." Nature 602, no. 7896 (2022): 223-228 [herein “Wurman”].
As per claim 8, Sharma does not specifically teach “wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream, winning a race, recovering from going off-course, and racing incidents.”
However, in the same field of endeavor namely training a reinforcement learning algorithm from an experience replay buffer, Wurman teaches “wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream, winning a race, recovering from going off-course, and racing incidents.” (Pg. 224, “we describe how we used model-free, off-policy deep RL to build a champion-level racing agent, which we call Gran Turismo Sophy (GT Sophy). GT Sophy was developed to compete with the world’s best players of the highly realistic PlayStation 4 (PS4) game Gran Turismo (GT) Sport” [© 2019 Sony Interactive Entertainment Inc] [in a car racing simulator]. Pg. 226, “the opportunities to learn certain skills are rare. We call this the exposure problem; certain states of the world are not accessible to the agent without the ‘cooperation’ of its opponents. For example, to execute a slingshot pass, a car must be in the slipstream of an opponent on a long straightaway, a condition that may occur naturally a few times or not at all in an entire race. If that opponent always drives only on the right, the agent will learn to pass only on the left and would be easily foiled by a human who chose to drive on the left. To address this issue, we developed a process that we called mixed-scenario training. We worked with a retired competitive GT driver to identify a small number of race situations that were probably pivotal on each track” [wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream]. Pg. 229, “The reward function was a hand-tuned linear combination of reward components computed on the transition between the previous state s and current state s′. The reward components were: course progress (Rcp), off-course penalty (Rsoc or Rloc), wall penalty (Rw), tyre-slip penalty (Rts), passing bonus (Rps), any-collision penalty (Rc), rear-end penalty (Rr) and unsporting-collision penalty (Ruc)” [recovering from going off-course, and racing incidents]. Pg. 227, “One of the advantages of using deep RL to develop a racing agent is that it eliminates the need for engineers to program how and when to execute the skills needed to win the race” [winning a race]. Further see Pgs. 224, 226, 227, and 229. The examiner has interpreted that developing a reinforcement learning racing agent in a car racing game that learns states such as slighshotting, off-course penalty, and collision penalty, to win a race as wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream, winning a race, recovering from going off-course, and racing incidents.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to add “wherein, in a car racing simulator, the event conditions are based on drafting in a car’s slipstream, winning a race, recovering from going off-course, and racing incidents” as conceptually seen from the teaching of Wurman, into that of Sharma because this modification of including car racing states for the advantageous purpose of considering advanced sports tactics in teaching computers competitive tasks (Wurman, Pg. 227). Further motivation to combine be that Sharma and Wurman are analogous art to the current claim directed to training a reinforcement learning algorithm from an experience replay buffer.
Claim 9 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Luo, Jieliang, and Hui Li. "Dynamic experience replay." In Conference on robot learning, pp. 1191-1200. PMLR, 2020 [herein “Luo”].
As per claim 9, Sharma does not specifically teach “wherein, in continuous control problems that already have rich reward signals, event conditions are based on exceeding thresholds in immediate rewards from a state.”
However, in the same field of endeavor namely creating custom experience replay buffers for training reinforcement agents, Luo teaches “wherein, in continuous control problems that already have rich reward signals, event conditions are based on exceeding thresholds in immediate rewards from a state”. (Pg. 2 Sect. 2, “the equation can be solved by model free RL algorithms to avoid using dynamics. DDPG is a model-free off-policy RL algorithm for continuous action spaces. In DDPG, an actor policy π : S → A is created to explore the space and store the collected transition (sj, aj ,sj+1, rj) in a replay buffer R” [in a continuous control problem]. Pg. 4, Sect. 3, “During training, all successful transitions that are generated by workers are saved in a pool, which is sampled periodically by each replay buffer and stored in the demonstration zone” [e.g., event conditions are based on rewards from a state]. Pg. 4 Sect. 3, “We use a simple linear reward function based on the distance between the goal pose and the current pose of the timber piece attached to the robot arm for both tasks. Additionally we use a large positive reward (+1000 for the peg-in-hole and +100 for the lap-joint) if the object is within a small distance of the goal pose where x is the current pose of the object, g is the goal pose, ε is a distance threshold, and R is the large positive reward. We use negative distance as our reward function to discourage the behavior of loitering around the goal because the negative distance also contains time penalty” [wherein, in continuous control problems that already have rich reward signals]. Further see Sect. 2-3. The examiner has interpreted that saving successful transitions into a replay buffer for peg-in-hole and lap-joint tasks that contain large positive rewards in a continuous action space as wherein, in continuous control problems that already have rich reward signals, event conditions are based on exceeding thresholds in immediate rewards from a state.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to add “wherein, in continuous control problems that already have rich reward signals, event conditions are based on exceeding thresholds in immediate rewards from a state” as conceptually seen from the teaching of Luo, into that of Sharma because this modification of filtering events for a continuous control problem for the advantageous purpose of efficiently training RL agents on successful samples (Luo, Pg. 1 & 3). Further motivation to combine be that Sharma and Luo are analogous art to the current claim directed creating custom experience replay buffers for training reinforcement agents.
Claim 13 is rejected under 35 U.S.C. § 103 as being unpatentable over Sharma in view of Daley, Brett, Cameron Hickert, and Christopher Amato. "Stratified experience replay: Correcting multiplicity bias in off-policy reinforcement learning." arXiv preprint arXiv:2102.11319 (2021) [herein “Daley”].
As per claim 13, Sharma does not specifically teach “determining a bias correction term for stochastic environments.”
However, in the same field of endeavor namely training reinforcement learning using experience relay, Daley teaches “determining a bias correction term for stochastic environments.” (Sect. 3, “Dividing this by Pr(𝑠, 𝑎) to eliminate the multiplicity bias, and then normalizing to make the probabilities sum to 1 over the set S×A×S, we arrive at the ideal sampling distribution: Pr(s’ | 𝑠, 𝑎) / |S × A|. Remarkably, this indicates that we can sample from two uniform distributions in succession to counter multiplicity b” and Sect 4, “SER offers a theoretically well-motivated alternative to the uniform distribution for off-policy deep RL methods. By correcting for multiplicity bias, SER helps agents learn significantly faster in small MDPs” [determining a bias correction term for stochastic environments]. Further see Sect. 3-4. The examiner has interpreted that creating an ideal sampling distribution by dividing by an additional factor in a Markov Decision Process as determining a bias correction term for stochastic environments.)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to add “determining a bias correction term for stochastic environments” as conceptually seen from the teaching of Daley, into that of Sharma because this modification of adding a bias correction for the advantageous purpose of correcting bias and training the reinforcement learning faster (Daley, Sect. 4). Further motivation to combine be that Sharma and Daley are analogous art to the current claim directed to training reinforcement learning using experience relay.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
de Bruin, Tim, Jens Kober, Karl Tuyls, and Robert Babuška. "The importance of experience replay database composition in deep reinforcement learning." In Deep reinforcement learning workshop, NIPS. 2015 teaches using a Deep Deterministic Policy Gradient to control a robot over a continuous action space using mini batch updates from an experience replay data table.
Cai, Qingpeng, Can Cui, Yiyuan Xiong, Wei Wang, Zhongle Xie, and Meihui Zhang. "A survey on deep reinforcement learning for data processing and analytics." IEEE Transactions on Knowledge and Data Engineering 35, no. 5 (2022): 4446-4465 teaches using samples from an experience replay to training a reinforcement learning algorithm in a Markov Decision Process.
Jaritz, Maximilian, Raoul De Charette, Marin Toromanoff, Etienne Perot, and Fawzi Nashashibi. "End-to-end race driving with deep reinforcement learning." In 2018 IEEE international conference on robotics and automation (ICRA), pp. 2070-2075. IEEE, 2018 teaches a method for using reinforcement learning to perform end-to-end drive for car racing using a asynchronous actor critic framework to learn car controls.
Examiner’s Note: The examiner has cited particular columns and line numbers in the reference that applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. In the case of amending the claimed invention, the applicant is respectfully requested to indicate the portion(s) of the specification which dictate(s) the structure relied on for the proper interpretation and also to verify and ascertain the metes and bound of the claimed invention.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Simeon P Drapeau whose telephone number is (571)-272-1173. The examiner can normally be reached Monday - Friday, 8 a.m. - 5 p.m. ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached on (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIMEON P DRAPEAU/Examiner, Art Unit 2188
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Prosecution Insights