DETAILED ACTION
This action is in response to the application filed 06/02/2023. Claims 1-20 are pending and have been examined.
EXAMINER NOTE
This action is being remailed in order to include references Koh et al. (Understanding Black-box Predictions via Influence Functions) and Zahavy et al. (Graying the black box: Understanding DQNs), which were previously cited as a relevant reference not relied on in the conclusion section of the previous mailing. There references are added in the 892 and are attached to herewith.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 rejected under 35 U.S.C. 101 because they are directed to an abstract idea that does not amount to significantly more.
Regarding Claim 1:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
generating, utilizing a clustering algorithm, trajectory clusters from trajectories utilized to train a reinforcement learning agent (this limitation is a mathematical because an equation is given within the specification (see paragraph 0056, algorithm 2). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 2).
generating a complementary target data set by removing a target trajectory cluster from the trajectory clusters (this limitation is a mental process as it encompasses a human mentally taking out a subset of data from a dataset).
generating a cluster attribution for the reinforcement learning agent by comparing a result of the test reinforcement learning agent and a result of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculate a cluster attribution with the equation that was given within the specification (see paragraph 0073, algorithm 5).
Therefore, claim 1 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
training a test reinforcement learning agent utilizing the complementary target data set (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 1 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
training a test reinforcement learning agent utilizing the complementary target data set is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 1 is subject-matter ineligible.
Regarding Claim 2:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 2 recites
generating a plurality of complementary target data sets by individually removing target trajectory clusters for the plurality of complementary target data sets (this limitation is a mental process as it encompasses a human mentally taking out a subset of data from a dataset).
Therefore, claim 2 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 2 further recites additional elements of
training test reinforcement learning agents utilizing the plurality of complementary target data sets (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 1 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 2 do not provide significantly more than the abstract idea itself, taken alone and in combination because
training test reinforcement learning agents utilizing the plurality of complementary target data sets is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 2 is subject-matter ineligible.
Regarding Claim 3:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
determining distances within a feature space between the plurality of complementary target data sets (this limitation is a mental process as it encompasses a human mentally calculating the distance with the equation that was given within the specification (see paragraph 0073, algorithm 5)).
selecting the cluster attribution based on the distances (this limitation is a mental process as it encompasses a human mentally choosing the cluster attribution).
Therefore, claim 3 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 3 does not further recite any additional elements, claim 3 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 3 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 3 is subject-matter ineligible.
Regarding Claim 4:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 4 recites
generating, utilizing a non-linear function, a plurality of complementary target data embeddings from the plurality of complementary target data sets (this limitation is a mathematical because an equation is given within the specification (see paragraph 0058, algorithm 3). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 3).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (this limitation is a mathematical because an equation is given within the specification (see paragraph 0058, algorithm 3). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 3).
determining the distances between the plurality of complementary target data embeddings and the trajectory embedding (this limitation is a mental process as it encompasses a human mentally calculating the distance with the equation that was given within the specification (see paragraph 0073, algorithm 5)).
Therefore, claim 4 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 4 does not further recite any additional elements, claim 4 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 4 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 4 is subject-matter ineligible.
Regarding Claim 5:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 5 recites
generating the cluster attribution for the reinforcement learning agent comprises comparing reinforcement learning decisions of the test reinforcement learning agents with a reinforcement learning decision of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally compare two different learning decisions).
Therefore, claim 5 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 5 does not further recite any additional elements, claim 5 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 5 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 5 is subject-matter ineligible.
Regarding Claim 6:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 6 recites
determining action distances between the reinforcement learning decisions of the test reinforcement learning agents and the reinforcement learning decision of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculating the distance with the equation that was given within the specification (see paragraph 0073, algorithm 5)).
selecting the cluster attribution by comparing the action distances (this limitation is a mental process as it encompasses a human mentally choosing the cluster attribution based on the action distances).
Therefore, claim 6 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 6 does not further recite any additional elements, claim 6 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 6 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 6 is subject-matter ineligible.
Regarding Claim 7:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 7 is dependent on claim 1, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 1 is applied here. Therefore, claim 7 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 7 further recites additional elements of
determining the trajectories by identifying for a first trajectory an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action (this element does integrate the abstract idea into a practical integration because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 7 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 7 do not provide significantly more than the abstract idea itself, taken alone and in combination because
determining the trajectories by identifying for a first trajectory an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h)).
Therefore, claim 7 is subject-matter ineligible.
Regarding Claim 8:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
generating … trajectory representations by encoding the trajectories utilized to train the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally using encoding data to create trajectory representations).
Therefore, claim 8 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 8 further recites additional elements of
utilizing a sequence encoder (this element does integrate the abstract idea into a practical integration because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 8 does not integrate an abstract idea into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 8 do not provide significantly more than the abstract idea itself, taken alone and in combination because
utilizing a sequence encoder recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h)).
Therefore, claim 8 is subject-matter ineligible.
Regarding Claim 9:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a method and is directed to a process, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
generating the trajectory clusters comprises utilizing a clustering algorithm to generate the trajectory clusters from trajectory representations (this limitation is a mathematical because an equation is given within the specification (see paragraph 0056, algorithm 2). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 2).
Therefore, claim 9 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 9 does not further recite any additional elements, therefore claim 9 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 9 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 9 is subject-matter ineligible.
Regarding Claim 10:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a system and is directed to a machine, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 10 recites
generating a reinforcement learning decision utilizing a reinforcement learning agent rained from a plurality of trajectories (this limitation is a mental process as it encompasses a human mentally creating a learning decision).
determining, utilizing a clustering algorithm, trajectory clusters from the plurality of trajectories (this limitation is a mathematical because an equation is given within the specification (see paragraph 0056, algorithm 2). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 2).
generating a plurality of complementary target data set by individually removing target trajectory clusters for the plurality of complementary target data sets (this limitation is a mental process as it encompasses a human mentally taking out multiple subsets of data from a dataset).
generating a cluster attribution for the reinforcement learning agent by comparing reinforcement learning decisions of the test reinforcement learning agents with the reinforcement learning decisions of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculate a cluster attribution with the equation that was given within the specification (see paragraph 0073, algorithm 5).
Therefore, claim 10 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 10 further recites additional elements of
a memory component (this element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(b))).
one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising… (this element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(b))).
generating a reinforcement learning decision utilizing a reinforcement learning agent rained from a plurality of trajectories (this element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(b))).
training a test reinforcement learning agent utilizing the plurality of complementary target data sets (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 10 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 10 do not provide significantly more than the abstract idea itself, taken alone and in combination because
a memory component uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(b)).
one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising… uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(b)).
generating a reinforcement learning decision utilizing a reinforcement learning agent rained from a plurality of trajectories uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(b)).
training a test reinforcement learning agent utilizing the plurality of complementary target data sets is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 10 is subject-matter ineligible.
Regarding Claim 11:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a system and is directed to a machine, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 11 recites
determining action distances within a feature space between the reinforcement learning decisions of the test reinforcement learning agents and the reinforcement learning decisions of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculating the distance with the equation that was given within the specification (see paragraph 0073, algorithm 5)).
Comparing the action distances to select a trajectory cluster for the cluster attribution (this limitation is a mental process as it encompasses a human mentally choosing the cluster attribution based on the action distances).
Therefore, claim 11 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 11 does not further recite any additional elements, claim 11 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 11 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 11 is subject-matter ineligible.
Regarding Claim 12:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a system and is directed to a machine, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 12 recites
generating complementary target embeddings from the plurality of complementary target data sets (this limitation is a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 3 from paragraph 0058).
generating a trajectory embedding from the plurality of trajectories utilized to train the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 3 from paragraph 0058).
Therefore, claim 12 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 12 does not further recite any additional elements, claim 12 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 12 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 12 is subject-matter ineligible.
Regarding Claim 13:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a system and is directed to a machine, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 13 recites
generating the cluster attribution for the reinforcement learning agent further comprises comparing the complementary target embeddings and the trajectory embedding (this limitation is a mental process as it encompasses a human mentally compare two different embeddings).
Therefore, claim 13 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 13 does not further recite any additional elements, claim 13 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 13 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 13 is subject-matter ineligible.
Regarding Claim 14:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a system and is directed to a machine, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 14 is dependent on claim 10, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 10 is applied here. Therefore, claim 14 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 14 further recites additional elements of
determining a trajectory from the plurality of trajectories by receiving an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action (this element does integrate the abstract idea into a practical integration because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 14 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 14 do not provide significantly more than the abstract idea itself, taken alone and in combination because
determining a trajectory from the plurality of trajectories by receiving an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h)).
Therefore, claim 14 is subject-matter ineligible.
Regarding Claim 15:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 15 recites
generating, utilizing a clustering algorithm, trajectory clusters from trajectories utilized to train a reinforcement learning agent (this limitation is a mathematical because an equation is given within the specification (see paragraph 0056, algorithm 2). This limitation could also be a mental process as it encompasses a human mentally creating trajectory clusters by using algorithm 2).
generating a complementary target data set by removing a target trajectory cluster from the trajectory clusters (this limitation is a mental process as it encompasses a human mentally taking out a subset of data from a dataset).
generating a cluster attribution for the reinforcement learning agent by comparing a result of the test reinforcement learning agent and a result of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculate a cluster attribution with the equation that was given within the specification (see paragraph 0073, algorithm 5).
Therefore, claim 15 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 15 further recites additional elements of
a non-transitory computer readable medium storing executable instructions which, when executed by a processing device cause the processing device to perform operations (this element does not integrate the abstract idea into a practical application because it recites a generic computing component on which to perform the abstract idea (see MPEP 2106.05(b))).
training a test reinforcement learning agent utilizing the complementary target data set (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 15 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 15 do not provide significantly more than the abstract idea itself, taken alone and in combination because
a non-transitory computer readable medium storing executable instructions which, when executed by a processing device cause the processing device to perform operations uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(b)).
training a test reinforcement learning agent utilizing the complementary target data set is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 15 is subject-matter ineligible.
Regarding Claim 16:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 16 recites
generating a plurality of complementary target data sets by individually removing target trajectory clusters for the plurality of complementary target data sets (this limitation is a mental process as it encompasses a human mentally taking out a subset of data from a dataset).
Therefore, claim 16 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 16 does not further recite any additional elements, claim 16 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B
Since there are no additional elements, claim 16 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 16 is subject-matter ineligible.
Regarding Claim 17:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 17 recites
generating the cluster attribution for the reinforcement learning agent by comparing a plurality of results of the test reinforcement learning agent and the result of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally compare two different results).
Therefore, claim 17 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 17 further recites additional elements of
training test reinforcement learning agents utilizing the plurality of complementary target data sets (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 17 is not integrated into a practical application
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 17 do not provide significantly more than the abstract idea itself, taken alone and in combination because
training test reinforcement learning agents utilizing the plurality of complementary target data sets is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Regarding Claim 18:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 18 recites
determining distances within a feature space between the complementary target data sets and the trajectories utilized to train the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally calculating the distance with the equation that was given within the specification (see paragraph 0073, algorithm 5)).
Therefore, claim 18 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 18 does not further recite any additional elements, claim 3 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 18 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 18 is subject-matter ineligible.
Regarding Claim 19:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 19 recites
generating, utilizing a non-linear function, complementary target data embeddings from the complementary target data sets (this limitation is a mathematical because an equation is given within the specification (see paragraph 0058, algorithm 3). This limitation could also be a mental process as it encompasses a human mentally creating data embeddings by using algorithm 3).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (this limitation is a mathematical because an equation is given within the specification (see paragraph 0058, algorithm 3). This limitation could also be a mental process as it encompasses a human mentally creating trajectory embeddings by using algorithm 3).
Therefore, claim 19 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 19 does not further recite any additional elements, claim 19 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 19 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 19 is subject-matter ineligible.
Regarding claim 20:
Subject Matter of Eligibility Analysis Step 1:
The claim recites a computer readable medium and is directed to a manufacture, which is one of the statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 20 recites
comparing the result of the test reinforcement learning agent and the result of the reinforcement learning agent (this limitation is a mental process as it encompasses a human mentally comparing the results of the learning agents).
comparing the distances to select a trajectory cluster for the cluster attribution (this limitation is a mental process as it encompasses a human mentally choosing a cluster based on the comparison of the distance.
Therefore, claim 20 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 20 does not further recite any additional elements, claim 20 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 20 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 20 is subject-matter ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 2, 5, 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghorbani et al.(Data Shapley: Equitable Valuation of Data for Machine Learning) (hereafter referred to as Ghorbani) in view of Lv et al. (Opponent modeling with trajectory representation clustering) (hereafter referred to as Lv).
Regarding claim 1, Ghorbani teaches
generating a complementary target data set by removing a target trajectory cluster from the trajectory clusters (Ghorbani, page 2-3, Overview section, “Let D = {(xi , yi)} n 1 be our fixed training set. D comes from n different sources of data where (xi , yi) is the i’th source…” and “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A . For each data source (xi, yi) ∈ D, our goal is to compute its data value, denoted by φi(D,A ,V) ∈ R, as a function of the three ingredients D,A and V. We will often write it as φi(V) or just φi to simplify notation. We simplify the notation even more for S and D to their set of indices—i.e. i ∈ S ⊆ D is the same as (xi , yi) is in S ⊆ D and therefore D = {1,...,n}.)” Examiner notes that Data Shapley is a value that determines contributions between data subsets. S is a subset of training set D, which means S is a data set that has been removed from D.
training a test reinforcement learning agent utilizing the complementary target data set (Ghorbani, page 3, Overview section, “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A”. Examiner notes that V(S) is the output of training S on the learning algorithm A, so we can interpret that train data S was trained on a reinforcement learning agent).
generating a cluster attribution for the reinforcement learning agent by comparing a result of the test reinforcement learning agent and a result of the reinforcement learning agent (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that V(S∪ {i})−V(S) is the comparison between the outputs of the subsets of training data S and the training data D).
Ghorbani does not teach, but Lv does teach
generating, utilizing a clustering algorithm, trajectory clusters from trajectories utilized to train a reinforcement learning agent (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to compare trajectory clusters from Lv. One of the ordinary skill in the art would have known to substitute a general type of data from Ghorbani to a more specific type of data, such as trajectory clusters from Lv. Therefore, using Lv’s trajectory clusters would have yielded the predictable results of comparing a subset of trajectory clusters with the whole set of trajectory clusters (See MPEP 2141 (III)(B) Simple substitution of one known element for another to obtain predictable results.
Regarding claim 2, Ghorbani further teaches
generating a plurality of complementary target data sets by individually removing target trajectory clusters for the plurality of complementary target data sets and training test reinforcement learning agents utilizing the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created and trained using a reinforcement learning agent).
Regarding claim 5, Ghorbani and Lv teaches the method of claims 1 and 2, Ghorbani further teaches
generating the cluster attribution for the reinforcement learning agent comprises comparing reinforcement learning decisions of the test reinforcement learning agents with a reinforcement learning decision of the reinforcement learning agent (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that V(S∪ {i})−V(S) is the comparison between the outputs of the subsets of training data S and the training data D).
Regarding claim 7, Ghorbani and Lv teaches the method of claim 1, Lv further teaches
determining the trajectories by identifying for a first trajectory an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action (Lv, Section 3.1 Problem formulation, “We describe the problem as a partially-observable stochastic game (POSG) composed of a finite set …, a state space …, the joint action space …, the joint observation space …, a transition function … denoting the transition probabilities between two states when given a joint action, and a reward function … for each agent”).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to use the structure of Lv’s trajectory data. One of the ordinary skill in the art would have known to apply Lv’s technique of trajectory data structure to Ghorbani. Therefore, applying Lv’s technique would have yielded the predictable results of a trajectory data comprising of an observed state, an action state, and a reward(See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 8, Ghorbani and Lv teaches the method of claim 1, Lv further teaches
generating, utilizing a sequence encoder, trajectory representations by encoding the trajectories utilized to train the reinforcement learning agent (Lv, page 169, Introduction section, 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level” and “Contrastive learning, as the most popular self-supervised learning algorithm in recent years, is different from generative encoding algorithms. Contrastive learning focuses on learning common features between similar instances and distinguishing differences between non-similar instances. van den Oord et al. first proposed InfoNCE loss, which encodes time-series data” (Lv, page 170, 2.2 Contrastive Learning section). Examiner notes that contrastive learning encodes timeseries data, which implies that there is a sequence encoder).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to use contrastive learning from Lv. One of the ordinary skill in the art would have known to apply Lv’s technique of contrastive learning to generate trajectory data to Ghorbani for comparison between trajectory data. Therefore, applying Lv’s technique would have yielded the predictable results of using a sequence encoder to create trajectory representations (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 9, Ghorbani and Lv teaches the method of claim 1, Lv further teaches
generating the trajectory clusters comprises utilizing a clustering algorithm to generate the trajectory clusters from trajectory representations (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to use trajectory representation clustering (TRC) from Lv. One of the ordinary skill in the art would have known to apply Lv’s technique of TRC to Ghorbani for comparison between trajectory data. Therefore, applying Lv’s technique would have yielded the predictable results of using TRC to create trajectory clusters(See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Claim(s) 3, 4, and 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghorbani and Lv in view of Agarwal et al. (Contrastive Behavioral Similarity Embeddings for Gernalization in Reinforcement Learning) (hereafter referred to as Agarwal)
Regarding claim 3, Ghorbani and Lv teaches the methods of claims 1 and 2, Agarwal teaches
determining distances within a feature space between the plurality of complementary target data sets and the trajectories utilized to train the reinforcement learning agent (Agarwal, page 1, Introduction section, 2nd paragraph, “Our approach exploits the fact that an agent, when operating in environments with similar underlying mechanics, exhibits at least short sequences of behaviors that are similar across these environments. Concretely, the agent is optimized to learn an embedding in which states are close when the agent’s optimal policies...are similar” and “We replace the absolute reward difference by a probability pseudometric between policies, denoted DIST...The DIST term captures the difference in local optimal behavior (A) while W1 captures long-term optimal behavior difference (B)...DIST is the total variation distance (T V ) when A is discrete, and we use the `1 distance between the mean actions of the two policies when A is continuous.” (Agarwal, page 3, Policy Similarity Metric section, 3rd paragraph). Examiner notes that “their approach” is referring to the Policy Similarity Metric or PSM. The embeddings are being mapped to the feature space and DIST is mapped to determining the distance.
selecting the cluster attribution based on the distances (Agarwal, page 2, Preliminaries section, 6th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”. Examiner notes that SimCLR is calculating the similarity, which is being mapped to the distance, between two states).
Ghorbani, Lv, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani and Lv to determine trajectory attributions by using the PSM from Agarwal. One of the ordinary skill in the art would have known to apply Agarwal’s technique of data attribution to the distance between trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of finding similarities between clusters (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 4, Ghorbani and Lv teaches the methods of claims 1, 2, and 3, Ghorbani and Lv further teach
generating, utilizing a non-linear function, a plurality of complementary target data embeddings from the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created and trained using a reinforcement learning agent).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv does not teach, but Agarwal teaches
generating, utilizing a non-linear function, a plurality of complementary target data embeddings from the plurality of complementary target data sets (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
determining the distances between the plurality of complementary target data embeddings and the trajectory embedding (Agarwal, page 3, Preliminaries section, 5th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”. Exmainer notes that the similarity function is mapped to determining the distance).
Ghorbani, Lv, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani and Lv to create an embedding space for the subset of trajectory data and the trajectory data as a whole with a non-linear function. One of the ordinary skill in the art would have known to apply Agarwal’s technique of generating an embedding space for the trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of creating an embedded space for trajectory clusters to find similarities (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 6, Ghorbani and Lv teaches the methods of claims 1, 2, and 5, Agarwal further teaches
comparing the reinforcement learning decisions of the test reinforcement learning agents with the reinforcement learning decision of the reinforcement learning agent further comprises: determining action distances between the reinforcement learning decisions of the test reinforcement learning agents and the reinforcement learning decision of the reinforcement learning agent (Agarwal, page 1, Introduction, 3rd paragraph, “the agent [Policy Similarity Metric] is optimized to learn an embedding in which states are close when the agent’s optimal policies in these states and future states are similar. This notion of proximity is general and it is applicable to observations from different environments”)
and selecting the cluster attribution by comparing the action distances (Agarwal, page 2, Preliminaries section, 6th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”. Examiner notes that SimCLR is calculating the similarity, which is being mapped to the distance, between two states).
Ghorbani, Lv, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, to apply Agarwal’s policy similarity metric to compare the action distances of the trajectory datasets from Ghorbani and Lv. One of the ordinary skill in the art would have known to apply Agarwal’s technique of similarity metrics and attribution to trajectory clusters. Therefore, applying Agarwal’s technique would have yielded predictable results of finding the similarities between clusters and finding the closest one (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Claim(s) 10, 14-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghorbani and Lv in view of Shim et al. (US 20210383158 A1) (hereafter referred to as Shim).
Regarding claim 10, Ghorbani teaches
generating a plurality of complementary target data sets by individually removing target trajectory clusters for the plurality of complementary target data sets (Ghorbani, page 2-3, Overview section, “Let D = {(xi , yi)} n 1 be our fixed training set. D comes from n different sources of data where (xi , yi) is the i’th source…” and “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A . For each data source (xi, yi) ∈ D, our goal is to compute its data value, denoted by φi(D,A ,V) ∈ R, as a function of the three ingredients D,A and V. We will often write it as φi(V) or just φi to simplify notation. We simplify the notation even more for S and D to their set of indices—i.e. i ∈ S ⊆ D is the same as (xi , yi) is in S ⊆ D and therefore D = {1,...,n}.)” Examiner notes that Data Shapley is a value that determines contributions between data subsets. S is a subset of training set D, which means S is a data set that has been removed from D.
training test reinforcement learning agents utilizing the plurality of complementary target data sets (Ghorbani, page 3, Overview section, “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A”. Examiner notes that V(S) is the output of training S on the learning algorithm A, so we can interpret that train data S was trained on a reinforcement learning agent).
generating a cluster attribution for the reinforcement learning agent by comparing reinforcement learning decisions of the test reinforcement learning agents with the reinforcement learning decision of the reinforcement learning agent (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that V(S∪ {i})−V(S) is the comparison between the outputs of the subsets of training data S and the training data D).
Ghorbani does not teach, but Lv does teach
generating a reinforcement learning decision utilizing a reinforcement learning agent trained from a plurality of trajectories (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments)”. Examiner notes that and reinforcement learning algorithm can be used, which means a learning decision is also created).
determining, utilizing a clustering algorithm, trajectory clusters from the plurality of trajectories (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to compare trajectory clusters from Lv. One of the ordinary skill in the art would have known to substitute a general type of data from Ghorbani to a more specific type of data, such as trajectory clusters from Lv. Therefore, using Lv’s trajectory clusters would have yielded the predictable results of comparing a subset of trajectory clusters with the whole set of trajectory clusters (See MPEP 2141 (III)(B) Simple substitution of one known element for another to obtain predictable results.
Ghorbani and Lv does not teach, but Shim does teach
a memory component (Shim, page 4, paragraph 0057, “The memory 220 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices”).
one or more processing devices coupled to the memory component, the one or more processing devices to perform operations (Shim, page 4, paragraph 0057, “The memory 220 optionally includes one or more storage devices remotely located from the one or more processing units 202”).
Ghorbani, Lv, and Shim are considered analogous to the claimed invention because they all deal with using the shapley value. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani and Lv to include a memory component and one more processing devices from Shim. One of the ordinary skill in the art would have known to apply Agarwal’s technique of using a memory component and one or more processing devices to perform Ghorbani and Lv’s technique. Therefore, applying Shim’s technique would have yielded predictable results (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 14, Ghorbani, Lv, and Shim teaches the system of claim 10, Lv further teaches
determining the trajectories by identifying for a first trajectory an observed state of a computing device, an action corresponding to the observed state, and a reward upon pursuing the action (Lv, Section 3.1 Problem formulation, “We describe the problem as a partially-observable stochastic game (POSG) composed of a finite set …, a state space …, the joint action space …, the joint observation space …, a transition function … denoting the transition probabilities between two states when given a joint action, and a reward function … for each agent”).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to use the structure of Lv’s trajectory data. One of the ordinary skill in the art would have known to apply Lv’s technique of trajectory data structure to Ghorbani. Therefore, applying Lv’s technique would have yielded the predictable results of a trajectory data comprising of an observed state, an action state, and a reward(See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 15, Ghorbani teaches
generating a complementary target data set by removing a target trajectory cluster from the trajectory clusters (Ghorbani, page 2-3, Overview section, “Let D = {(xi , yi)} n 1 be our fixed training set. D comes from n different sources of data where (xi , yi) is the i’th source…” and “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A . For each data source (xi, yi) ∈ D, our goal is to compute its data value, denoted by φi(D,A ,V) ∈ R, as a function of the three ingredients D,A and V. We will often write it as φi(V) or just φi to simplify notation. We simplify the notation even more for S and D to their set of indices—i.e. i ∈ S ⊆ D is the same as (xi , yi) is in S ⊆ D and therefore D = {1,...,n}.)” Examiner notes that Data Shapley is a value that determines contributions between data subsets. S is a subset of training set D, which means S is a data set that has been removed from D.
training a test reinforcement learning agent utilizing the complementary target data set (Ghorbani, page 3, Overview section, “We write V(S,A ), or just V(S) for short, to denote the performance score of the predictor trained on train data S using the learning algorithm A”. Examiner notes that V(S) is the output of training S on the learning algorithm A, so we can interpret that train data S was trained on a reinforcement learning agent).
generating a cluster attribution for the reinforcement learning agent by comparing a result of the test reinforcement learning agent and a result of the reinforcement learning agent (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that V(S∪ {i})−V(S) is the comparison between the outputs of the subsets of training data S and the training data D)
Ghorbani does not teach, but Lv does teach
generating, utilizing a clustering algorithm, trajectory clusters from trajectories utilized to train a reinforcement learning agent (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv are considered analogous to the claimed invention because they both deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani to compare trajectory clusters from Lv. One of the ordinary skill in the art would have known to substitute a general type of data from Ghorbani to a more specific type of data, such as trajectory clusters from Lv. Therefore, using Lv’s trajectory clusters would have yielded the predictable results of comparing a subset of trajectory clusters with the whole set of trajectory clusters (See MPEP 2141 (III)(B) Simple substitution of one known element for another to obtain predictable results.
Ghorbani and Lv does not teach, but Shim does teach
A non-transitory computer readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations (Agarwal, page 4, paragraph 0057, “The memory 220 comprises a non-transitory computer readable storage medium. In some implementations, the memory 220 or the non-transitory computer readable storage medium of the memory 220 stores the following programs, modules and data structures, or a subset thereof including an optional operating system 230 and a training engine 240”).
Ghorbani, Lv, and Shim are considered analogous to the claimed invention because they all deal with using the shapley value. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani and Lv to include a non-transitory computer readable medium from Shim. One of the ordinary skill in the art would have known to apply Agarwal’s technique of using a non-transitory computer readable medium to perform Ghorbani and Lv’s technique. Therefore, applying Shim’s technique would have yielded predictable results (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 16, Ghorbani, Lv, and Shim teach the non-transitory computer readable medium of claim 15, Ghorbani further teaches
generating a plurality of complementary target data sets by individually removing target trajectory clusters for the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created).
Regarding claim 17, Ghorbani further teaches
training test reinforcement learning agents utilizing the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created and trained using a reinforcement learning agent).
generating the cluster attribution for the reinforcement learning agent by comparing a plurality results of the test reinforcement learning agent and a result of the reinforcement learning agent (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that V(S∪ {i})−V(S) is the comparison between the outputs of the subsets of training data S and the training data D).
Claim(s) 11-13, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghorbani, Lv, and Shim in view of Agarwal.
Regarding claim 11, Ghorbani, Lv, and Shim teaches the system of claim 10, Agarwal teaches
determining action distances within a feature space between the reinforcement learning decisions of the test reinforcement learning agents and the reinforcement learning decision of the reinforcement learning agent (Agarwal, page 1, Introduction section, 2nd paragraph, “Our approach exploits the fact that an agent, when operating in environments with similar underlying mechanics, exhibits at least short sequences of behaviors that are similar across these environments. Concretely, the agent is optimized to learn an embedding in which states are close when the agent’s optimal policies...are similar” and “the agent [Policy Similarity Metric] is optimized to learn an embedding in which states are close when the agent’s optimal policies in these states and future states are similar. This notion of proximity is general and it is applicable to observations from different environments” ( Agarwal, page 1, Introduction, 3rd paragraph) Examiner notes that “their approach” is referring to the Policy Similarity Metric or PSM. The embeddings are being mapped to the feature space).
comparing the action distances to select a trajectory cluster for the cluster attribution (Agarwal, page 2, Preliminaries section, 6th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”. Examiner notes that SimCLR is calculating the similarity, which is being mapped to the distance, between two states).
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to apply Agarwal’s policy similarity metric to compare the action distances of the trajectory datasets from Ghorbani and Lv. One of the ordinary skill in the art would have known to apply Agarwal’s technique of similarity metrics and attribution to trajectory clusters. Therefore, applying Agarwal’s technique would have yielded predictable results of finding the similarities between clusters and finding the closest one (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 12, Ghorbani, Lv, and Shim teaches the system of claim 10, Ghorbani and Lv further teach
generating complementary target data embeddings from the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created and trained using a reinforcement learning agent).
generating a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani, Lv, and Shim does not teach, but Agarwal teaches
generating complementary target data embeddings from the plurality of complementary target data sets (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
generating a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to create an embedding space from Agarwal for the subset of trajectory data and the trajectory data as a whole. One of the ordinary skill in the art would have known to apply Agarwal’s technique of generating an embedding space for the trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of creating an embedded space for the trajectory datasets (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 13, Ghorbani, Lv, Shim, and Agarwal teach the system of claims 10 and 12, Agarwal further teaches
generating the cluster attribution for the reinforcement learning agent further comprises comparing the complementary target embeddings and the trajectory embedding (Agarwal, page 3, Preliminaries section, 5th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”).
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to use SimCLR from Agarwal. One of the ordinary skill in the art would have known to apply Agarwal’s technique of SimCLR to compare embeddings of the trajectory subsets and the trajectory data as a whole of Ghorbani and Lv . Therefore, applying Agarwal’s technique would have yielded the predictable results of comparing the trajectory datasets (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 18, Ghorbani, Lv, and Shim teach the non-transitory computer readable medium of claim 15, Agarwal teaches
determining distances within a feature space between the complementary target data sets and the trajectories utilized to rain the reinforcement learning agent (Agarwal, page 1, Introduction section, 2nd paragraph, “Our approach exploits the fact that an agent, when operating in environments with similar underlying mechanics, exhibits at least short sequences of behaviors that are similar across these environments. Concretely, the agent is optimized to learn an embedding in which states are close when the agent’s optimal policies...are similar” and “We replace the absolute reward difference by a probability pseudometric between policies, denoted DIST...The DIST term captures the difference in local optimal behavior (A) while W1 captures long-term optimal behavior difference (B)...DIST is the total variation distance (T V ) when A is discrete, and we use the `1 distance between the mean actions of the two policies when A is continuous.” (Agarwal, page 3, Policy Similarity Metric section, 3rd paragraph). Examiner notes that “their approach” is referring to the Policy Similarity Metric or PSM. The embeddings are being mapped to the feature space and DIST is mapped to determining the distance.
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to determine the distance by using the PSM from Agarwal. One of the ordinary skill in the art would have known to apply Agarwal’s technique of data attribution to the distance between trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of determining the distance between the trajectory datasets (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 19, Ghorbani, Lv, Shim, and Agarwal teach the non-transitory computer readable medium of claim 15-18, Ghorbani and Lv teach
generating, utilizing a non-linear function, a plurality of complementary target data embeddings from the plurality of complementary target data sets (Ghorbani, page 3, Proposition section, “Any data valuation φ(D,A ,V) that satisfies properties 1-3 above must have the form
φi = C ∑ S⊆D−{i} V(S∪ {i})−V(S) n−1 |S|
where the sum is over all subsets of D not containing i and C is an arbitrary constant. We call φi the Data Shapley value of source i”. Examiner notes that the equation contains the sum of all subsets, which means there is a plurality of subsets that were created and trained using a reinforcement learning agent).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Lv, page 169, Introduction 4th paragraph, “to distinguish various trajectories, we self-supervise extracted policy representations from interactive trajectories by contrastive learning and clustering at the representational level. Our proposed method, trajectory representation clustering (TRC), can be combined with any existing reinforcement learning (RL) algorithm, to avoid policy forgetting in non-stationary multi-agent environments).
Ghorbani and Lv does not teach, but Agarwal teaches
generating, utilizing a non-linear function, a plurality of complementary target data embeddings from the plurality of complementary target data sets (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
generating, utilizing the non-linear function, a trajectory embedding from the trajectories utilized to train the reinforcement learning agent (Agarwal, page 2, Introduction, 5th paragraph, “We employ PSM for representation learning and introduce policy similarity embeddings (PSEs) for deep RL. To do so, we present a general contrastive procedure (Section 4) to learn an embedding based on any state similarity metric” and “For all our experiments, we use a single ReLU layer with k = 64 units for the non-linear projection to obtain the embedding zθ (Agarwal, page 21, Contrastive Embedding)).
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to create an embedding space for the subset of trajectory data and the trajectory data as a whole with a non-linear function. One of the ordinary skill in the art would have known to apply Agarwal’s technique of generating an embedding space for the trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of creating an embedded space for trajectory clusters to find similarities (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Regarding claim 20, Ghorbani, Lv, Shim, and Agarwal teach the non-transitory computer readable medium of claim 15-18, Agarwal further teaches
comparing the distances to select a trajectory cluster for the cluster attribution (Agarwal, page 2, Preliminaries section, 6th paragraph, “We adapt SimCLR (Chen et al., 2020), a popular contrastive method for learning embeddings of image inputs. Given two inputs x and y, their embedding similarity is sθ(x,y) = sim(zθ(x),zθ(y)), where sim(u,v) = uTv u v denotes the cosine similarity function”. Examiner notes that SimCLR is calculating the similarity, which is being mapped to the distance, between two states).
Ghorbani, Lv, Shim, and Agarwal are considered analogous to the claimed invention because they all deal with evaluating data in reinforcement learning. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Ghorbani, Lv, and Shim to use SimCLR from Agarwal. One of the ordinary skill in the art would have known to apply Agarwal’s technique of SimCLR to compare embeddings of the trajectory subsets and the trajectory data as a whole of Ghorbani and Lv. Therefore, applying Agarwal’s technique would have yielded the predictable results of comparing the trajectory datasets (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predictable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's
disclosure. Koh et al. (Understanding Black-box Predictions via Influence Functions) discloses influence functions to address the black box problem in reinforcement learning. Zahavy et al. (Graying the black box: Understanding DQNs) discloses the analysis of Deep Q-Networks to help solve black box problems.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN VO whose telephone number is (571)272-9622. The examiner can normally be reached Monday - Friday from 7-3 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.V./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights