DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application and the preliminary amendment filed 10/19/2023. In the amendment claims 1-6 and 10-12 were amended, and no claims were added or canceled. Thus, claims 1-12 are pending and have been examined.
Priority
The present application is a national stage entry (under 35 U.S.C. § 371) of international application no. PCT/JP2021/016630 filed 4/26/2021.
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed 10/19/2023, 5/30/2024 and 7/11/2024, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Specification
The disclosure is objected to because of the following informalities:
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required:
Claims 10-12 do not appear to have support in the originally filed specification. There does not appear to be any discussion of any “computer readable information recording medium” or “computer readable information recording” media. Paragraph 92 mentions “a non-transitory tangible medium” and paragraphs 104-106, which include the language of original claims 10-12, recite “A program storage medium” and “The program storage medium”. However, the specification fails to mention, let alone describe or discuss any “computer readable information recording medium” as recited in claims 10-12. Appropriate correction is required.
In paragraphs 4-8 of the specification in the “Background Art”, “Citation List Non patent Literature” and “Summary of Invention Technical Problem” sections of the specification, non-patent literature references are referred to (see, e.g., references to “Non patent literature 1 describes Maximum Entropy Inverse Reinforcement Learning (ME-IRL)”, “Non patent literature 2 also describes Guided Cost Learning (GCL)”, “NPL 1: RD. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforcement learning," In AAAI, AAAI '08, 2008.”, “NPL 2: Chelsea Finn, Sergey Levine, Pieter Abbeel, "Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization'', Proceedings of The 33rd International Conference on Machine Learning, PMLR 48, pp. 49-58, 2016.”, “in the ME-1RL described in Non patent literature 1, it is necessary to calculate the sum of rewards for all possible trajectories during training.” and “To address this issue, the GCL described in Non patent literature 2 calculates this value approximately by weighted sampling. Here, when using weighted. sampling with GCL, it is necessary to assume the distribution of the sampling itself However, there are some problems, such as combinatorial optimization problems, where it is not known how to set the sampling distribution, so the method described in Non patent literature 2 is not applicable to various mathematical optimization.”) The listing of references in the specification is not a proper information disclosure statement. 37 CFR 1.98(b) requires a list of all patents, publications, or other information submitted for consideration by the Office, and MPEP § 609.04(a) states, "the list may not be incorporated into the specification but must be submitted in a separate paper." Therefore, unless the references have been cited by the examiner on form PTO-892, they have not been considered. It is noted, however, that applicant appears to have furnished citations to and copies of the references where appropriate (i.e., for the above-noted non-patent literature references referred to in paragraphs 4-8) in the above-referenced information disclosure statement filed 10/19/2023 IDS. Appropriate correction is required.
Paragraphs 10-12, 67, 82, 95, 101, 104 and 107 recite “feature is set to satisfy Lipschitz continuity condition”. These recitations are grammatically incorrect and appear to be missing one or more words between “satisfy” and “Lipshitz”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” or “the”, as appropriate, between “satisfy” and “Lipshitz”. Appropriate correction is required.
Paragraphs 10, 11, 71, 82, 95, 101 and 104 recite “a trajectory that minimizes Wasserstein distance”. These recitations are grammatically incorrect and appear to be missing one or more words between “minimizes” and “Wasserstein”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” or “the”, as appropriate, between “minimizes” and “Wasserstein”. Appropriate correction is required.
Paragraphs 11, 12, 69, 71, 82, 101, 104 and 107 recite “log-likelihood of Boltzmann distribution”. These recitations are grammatically incorrect and appear to be missing one or more words between “of” and “Boltzmann”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” or “the”, as appropriate, between “of” and “Boltzmann”. Appropriate correction is required.
Paragraphs 96, 97, 102, 103, 105, 106, 108 and 109 recite “coefficient that attenuates degree”. These recitations are grammatically incorrect and appear to be missing one or more words between “attenuates” and “degree”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” or “the”, as appropriate, between “attenuates” and “degree”. Appropriate correction is required.
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
The abstract of the disclosure is objected to because the recitation of “The function input means 91 accepts input of a reward function whose feature is set to satisfy Lipschitz continuity condition. The estimation means 92 estimates a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function. The updating means 93 updates, based on the estimated trajectory, the parameter of the reward function to maximize the log--likelihood of Boltzmann distribution derived from a principle of a maximum entropy.” is grammatically incorrect. Each of the above sentences are grammatically incorrect and appear to be missing one or more words.
First, in the 1st sentence “The function input means 91 accepts input of a reward function whose feature is set to satisfy Lipschitz continuity condition” the element number 91 is unnecessary and there appears to be missing the word/article “a” between “satisfy” and “Lipshitz”.
Second, in the 2nd sentence “The estimation means 92 estimates a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function.” the element number 92 is unnecessary and there appears to be a missing article “a” between “minimizes” and “Wasserstein”.
Third, in the 3rd sentence “The updating means 93 updates, based on the estimated trajectory, the parameter of the reward function to maximize the log--likelihood of Boltzmann distribution derived from a principle of a maximum entropy.” the element number 93 is unnecessary and there appears to be a missing article “a” between “of” and “Boltzmann”.
A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claims 1-12 are objected to because of the following informalities:
Line 5 of independent claim 1, lines 2-3 of independent claim 7, and line 5 of independent claim 10 recite “feature is set to satisfy Lipschitz continuity condition”. These recitations are grammatically incorrect and appear to be missing one or more words between “satisfy” and “Lipshitz”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” between “satisfy” and “Lipshitz”. Appropriate correction is required.
Lines 6-7 of independent claim 1, line 4 of independent claim 7, and lines 7-8 of independent claim 10 recite “a trajectory that minimizes Wasserstein distance”. These recitations are grammatically incorrect and appear to be missing one or more words between “minimizes” and “Wasserstein”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” between “minimizes” and “Wasserstein”. Appropriate correction is required.
Lines 11-12 of independent claim 1, line 8 of independent claim 7, and lines 12-13 of independent claim 10 recite “log-likelihood of Boltzmann distribution”. These recitations are grammatically incorrect and appear to be missing one or more words between “of” and “Boltzmann”. If supported by applicant’s original specification, the examiner suggests that one way to address these objections would be to add the word/article “a” between “of” and “Boltzmann”. Appropriate correction is required.
Also, claims 2-6, 8-9 and 11-12, which depend directly or indirectly from claims 1, 7 and 10, respectively, are objected to based on their respective dependencies from claims 1, 7 and 10.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-12 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
The claims are replete with indefinite language including many terms that lack antecedent basis, many unclear terms, relative terms, and the other indefinite language detailed below.
Independent claims 1, 7 and 10 recite, using respective similar language, estimate a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function (see, lines 6-9 of claim 1, lines 4-6 of claim 7 and lines 7-10 of claim 10). These recitations are grammatically incorrect, appear to be missing words, and unclear.
First, the recitation “a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function” appears to be missing words/articles (i.e., “a” and/or “the”) between “minimizes” and “Wasserstein” (see above-noted objections to these claims) and between “represents” and “distance”.
Second, the recitation of “distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function” is unclear because it is unclear what “a trajectory of an expert” refers to (i.e., a trajectory of data from a subject matter expert, a trajectory of decisions/output from an expert system, or some other trajectory associated with the recited “an expert”?). Aside from mentioning “The storage unit 10 may store decision-making history data (trajectory) of an expert”, “the input unit 20 may accept input of the decision-making history data of an expert (specifically, state and action pairs)”, “The input unit 20 accepts input of expert data (i.e., trajectory/decision-making history data of an expert)” and “The training data storage unit 2200 may, for example, store decision-making history data of an expert.” in paragraphs 24, 26, 67 and 76, applicant’s specification merely repeats the claim language re: “a trajectory of an expert” (see, e.g., paragraphs 10-12, 71, 82, 95, 101, 103 and 107) without describing or defining what is meant by the term.
Lastly, it is unclear if the subsequently-recited “a trajectory determined” refers to the previously-introduced “a trajectory of an expert”, or to a wholly separate “trajectory”. In any event, as discussed above, it is unclear what is meant by “a trajectory of an expert”.
For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function” as any trajectory, history, trend or path that minimizes a Wasserstein distance between a probability distribution of a trajectory, direction, history, trend or path of decision-making data of an expert, such as, but not limited to state and action pairs, and a probability distribution of any trajectory, direction, history, trend or path determined based on a parameter of the reward function. Appropriate correction is required.
Independent claims 1, 7 and 10 each recite “a lower limit of the log-likelihood” (see, the last limitation of each of these claims). The term “a lower limit of the log-likelihood” in these claims is a relative term which renders the claims indefinite. The term “a lower limit of the log-likelihood” is not defined by the claims, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Aside from merely repeating the claim language (see, e.g., paragraphs 10-12, 104 and 107), the specification only provides examples in stating “the weight updating unit 60 derives the lower limit L_(ϴ) (L_ denotes the subscript underbar of L) in the distance measure between the distributions used in ME--IRL as in Equation 9 below. The derived equation is sometimes hereafter referred to simply as the lower limit of the log-likelihood” and “convergence decision unit 70 determines whether the lower limit of the log-likelihood has converged. The determination method is arbitrary. For example, the convergence decision unit 70 may determine that the distance measure between distributions has converged when the absolute value of the lower limit of the log-likelihood becomes smaller than a predetermined threshold value.” in paragraphs 38 and 60. Thus, it fails to describe or define what is meant by this term. As such, the specification does not provide a standard for determining the requisite degree, numeric value, percentage, threshold, or range of values for the claimed “a lower limit of the log-likelihood” in claims 1, 7 and 10. For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “a lower limit of the log-likelihood” as any lower bound, lower threshold value, lower cutoff value, minimum value, or lowest limit value of the log-likelihood. Appropriate correction is required.
Independent claims 1, 7 and 10 each recite “the average value of reward for the parameter” (see, the last limitation of each of these claims). There is insufficient antecedent basis for this limitation in these claims. Applicant did not previously introduce any “average value of reward for the parameter” in these claims. For examination purposes, “the average value of reward for the parameter” is being interpreted as any average value of reward for the parameter. Appropriate correction is required.
Claim 5 recites “an upper bound of a log sum exponential” (see, lines 3-4 of the claim). The term “an upper bound of a log sum exponential” in this claim is a relative term which renders the claim indefinite. The term “an upper bound of a log sum exponential” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Aside from merely repeating the claim language (see, e.g., paragraphs 88 and 99), the specification only provides an example in stating “updating, the weight updater 60 in this exemplary embodiment derives the upper limit of the log sum exponential (hereinafter referred to as logSumExp) from the second term in Equation 6 (i.e., the sum of the rewards for all trajectories)” in paragraph 38. These are the sole mentions of any “upper bound” or “upper limit” of any log sum exponential in applicant’s specification. Thus, the specification fails to describe or define what is meant by this term. As such, the specification does not provide a standard for determining the requisite degree, numeric value, threshold value, or range of values for the claimed “an upper bound of a log sum exponential” in claim 5. For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “an upper bound of a log sum exponential” as any upper bound value, maximum value, or highest limit value, cutoff or threshold of a log sum exponential. Appropriate correction is required.
Claims 2, 8 and 11 recite, using respective similar language, “an attenuation coefficient that attenuates degree to which the portion corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood”. These recitations are grammatically incorrect and unclear. First, “coefficient that attenuates degree” appears to be missing the word/article “a” between “attenuates” and “degree”. Second, there is insufficient antecedent basis for the limitation “the portion corresponding to the entropy regularization term” in these claims. Applicant did not previously introduce any “portion corresponding to the entropy regularization term” in these claims, or their respective base claims, claims 1, 7 and 10. Applicant previously introduced “an entropy regularization term” in claims 1, 7 and 10 (see, the last limitation of these claims). However, it is unclear whether the subsequent recitations of “the portion corresponding to the entropy regularization term” refer to a particular portion or subset of the previously-introduced “entropy regularization term”, or to a portion or subset of some other data that corresponds to or is associated or correlated with “the entropy regularization term”. For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “an attenuation coefficient that attenuates degree to which the portion corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood” as any coefficient that attenuates a degree to which any portion or subset of data corresponding to or associated or correlated with the previously-introduced entropy regularization term that contributes to maximizing the lower limit of the log-likelihood similarity. Appropriate correction is required.
Claims 3, 9 and 12 each recite, using respective similar language, “an attenuation coefficient that attenuates degree to which the entropy regularization term contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term”. These recitations are grammatically incorrect and unclear. First, “coefficient that attenuates degree” appears to be missing the word/article “a” between “attenuates” and “degree”. Second, there is insufficient antecedent basis for the limitation “the portion corresponding to the entropy regularization term” in these claims. Applicant did not previously introduce any “portion corresponding to the entropy regularization term” in these claims, or their respective base claims, claims 1, 7 and 10. Applicant previously introduced “an entropy regularization term” in claims 1, 7 and 10 (see, the last limitation of these claims). However, it is unclear whether the subsequent recitations of “the portion corresponding to the entropy regularization term” refer to a particular portion or subset of the previously-introduced “entropy regularization term”, or to a portion or subset of some other data that corresponds to or is associated or correlated with “the entropy regularization term”. For the purposes of determining patent eligibility and comparison with the prior art, the examiner is interpreting the term “an attenuation coefficient that attenuates degree to which the entropy regularization term contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term” as any coefficient that attenuates a degree to which any portion or subset of data corresponding to or associated or correlated with the previously-introduced entropy regularization term that contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term. Appropriate correction is required.
Also, claim 4, which depends directly from claim 3, is rejected under 35 U.S.C. 112(b) as being indefinite under the same rationale as claim 3.
Claim 4 recites “the moving average of the log-likelihood”. There is insufficient antecedent basis for this limitation in this claim. Applicant did not previously introduce any “moving average of the log-likelihood” in this claim, intervening claim 3, or in its base claim, claim 1. Applicant previously introduced “the average value of reward for the parameter” in claim 1. However, it is unclear if the subsequently-recited “the moving average of the log-likelihood” refers to a moving average of the previously-introduced “average value of reward for the parameter”, or to a moving average “of the log-likelihood”. For examination purposes, “the moving average of the log-likelihood” is being interpreted as any moving average of the log-likelihood. Appropriate correction is required.
Claim 5 recites “the lower bound for the log-likelihood”. There is insufficient antecedent basis for this limitation in this claim. Applicant did not previously introduce any “lower bound for the log-likelihood” in this claim or in its base claim, claim 1. Applicant previously introduced “the derived lower limit of the log-likelihood” in claim 1. However, it is unclear if the subsequently-recited “the lower bound for the log-likelihood” refers to the previously-introduced “derived lower limit of the log-likelihood”, some lower bound value or limit of the previously-introduced “derived lower limit of the log-likelihood”, or to a separate, lower bound value or limit for the log-likelihood. For examination purposes, “the lower bound for the log-likelihood” is being interpreted as any lower bound value or lower limit value for the previously-introduced log-likelihood. Appropriate correction is required.
Also, claims 2-6, 8-9 and 11-12, which depend directly or indirectly from claims 1, 7 and 10, respectively are rejected under 35 U.S.C. 112(b) as being indefinite under the same rationale as claims 1, 7 and 10.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis below of the claims’ subject matter eligibility follows the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”). and the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, 89 Fed. Reg. 58128-58138 (July 17, 2024) (“2024 AI SME Update”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself.
Regarding independent claims 1, 7 and 10 these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a device, corresponding to an article of manufacture, claim 7 is directed to a method, corresponding to a process, and claim 10 is directed to a non-transitory computer readable information recording medium, corresponding to an article of manufacture, which are each one of one of the four statutory categories of invention.
Step 2A Prong One Analysis: The claims are directed to an abstract idea. In particular, the claims recite mathematical concepts (including mathematical relationships, mathematical formulas or equations, and mathematical calculations). Claims 1, 7 and 10 recite, using respective similar language, inter alia: operations (claim 1) and method steps (claims 7 and 10) to:
estimate a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function1;
update, based on the estimated trajectory, the parameter of the reward function to maximize the log-likelihood of Boltzmann distribution derived from a principle of a maximum entropy; and
derive, as a lower limit of the log-likelihood, an expression for subtracting, from the Wasserstein distance, an entropy regularization term defined by an expression for the maximum reward value for the parameter minus the average value of reward for the parameter, and update the parameter of the reward function to maximize the derived lower limit of the log-likelihood.
The above-noted estimate, update and derive limitations, as drafted, are a process that, under its broadest reasonable interpretation (BRI), covers mathematical concepts (i.e., mathematical calculations to estimate a trajectory, history, trend or path of data values that minimizes a Wasserstein distance between a calculated probability distributions of a trajectory, direction, history, trend or path of expert data values and a probability distribution of a trajectory, direction, history, trend or path determined based on an observed parameter of a reward function, update the reward function parameter in order to maximize the log-likelihood of Boltzmann distribution, derive an expression for subtracting from the Wasserstein distance, an entropy regularization term defined by an expression for the maximum reward value for the parameter minus (i.e., subtraction) a calculated average value of reward for the parameter, and then update the reward function parameter to maximize the derived lower limit/bound of the log-likelihood, which are mathematical concepts.
Under their BRI, in light of the specification, the estimate, update and derive limitations encompass the mathematical concepts of calculating values, estimating trajectories of data values to minimize a Wasserstein distance based on an observed parameter of a reward function (i.e., a mathematical function), updating the reward function parameter to maximize the log-likelihood of Boltzmann distribution, deriving a mathematical expression for subtracting from the Wasserstein distance, and then updating the reward function parameter to maximize the derived lower limit/bound of the log-likelihood, as described in the specification in paragraphs 15-17, 28-29, 33-43, 46-52 and 57-59 (see, e.g., equations 1-14 in the Math. 1-10 sections of the specification).
If the claim limitations, under their broadest reasonable interpretations, cover mathematical relationships, mathematical formulas or equations, or mathematical calculations, then they fall within the “Mathematical Concepts” grouping of abstract ideas. See MPEP 2106.04(a)(2) § I. But for the recitation of generic computer components (i.e., the “learning device comprising: a memory storing instructions; and one or more processors configured to execute the instructions” of claim 1 and the “non-transitory computer readable information recording medium storing a learning program, when executed by a processor, that performs a method” of claim 10), the limitations of claims 1, 7 and 10, cover mathematical relationships, mathematical formulas or equations, and mathematical calculations. Accordingly, claims 1, 7 and 10 recite an abstract idea.
Therefore, the claims are directed to an abstract idea (mathematical concept).
Step 2A Prong Two Analysis: This judicial exception is not integrated into a practical application. Claims 1, 7 and 10 do not recite any additional limitations or elements which integrate the abstract idea into a practical application.
Claims 1, 7 and 10 recite, using respective similar language, the additional limitation of “accept input of a reward function whose feature is set to satisfy Lipschitz continuity condition”. The above-noted “accept input of a reward function” limitation merely recites the insignificant extra-solution activity of data gathering that does not integrate a judicial exception into a practical application. See MPEP 2106.05(g). That is, this limitation is adding insignificant extra-solution activity (amount to necessary data gathering) to the judicial exception, as discussed in MPEP § 2106.05(g).
Claims 1, 7 and 10 also recited the additional elements of “A learning device comprising: a memory storing instructions; and one or more processors configured to execute the instructions to:” <perform the above-noted estimate, update and derive operations> (claim 1), “A learning method for a computer comprising:” <performing the above-noted estimate, update and derive steps> (claim 7) and “A non-transitory computer readable information recording medium storing a learning program, when executed by a processor, that performs a method” (claim 10). These elements are recited at a high level of generality as mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea (i.e., as generic computer components performing generic computer functions). See MPEP 2106.05(f). That is, claims 1 and 10 are directed to generic computers and computing components performing mathematical functions.
In the context of claims 1, 7 and 10, the “learning device comprising: a memory storing instructions; and one or more processors configured to execute the instructions” of claim 1, the “learning method for a computer comprising” of claim 7 and the “non-transitory computer readable information recording medium storing a learning program, when executed by a processor, that performs a method” of claim 10 are considered to be mere instructions to apply the judicial exception (abstract idea).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception (i.e., the additional elements describe a device and medium for applying the abstract ideas). Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “learning device comprising: a memory storing instructions; and one or more processors configured to execute the instructions” of claim 1, “a computer” of claim 7 and the “non-transitory computer readable information recording medium storing a learning program, when executed by a processor, that performs a method” of claim 10) do not amount to significantly more than the judicial exception.
Moreover, receiving, communicating, and transmitting data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, the above-noted recitations of “accept input of a reward function whose feature is set to satisfy Lipschitz continuity condition” are the well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
For the same reasons as noted above, the recitation of a generic computer and generic computer components like a generic device with memory and processor components is not significantly more than the judicial exception. Nothing in the claims provide significantly more than this.
These claims do not recite any additional elements that integrate the abstract idea into a practical application or provides significantly more than the abstract idea, and thus, the claims are subject-matter ineligible.
Regarding claims 2, 8 and 11, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a device as depending from claim 1, claim 8 is directed to a method as depending from claim 7, and claim 11 is directed to a computer readable information recording medium as depending from claim 10, thus the analysis for patent eligibility of claims 1, 7 and 10, respectively, are incorporated herein.
Step 2A Prong 1: The claims each recite, using respective similar language, “set, to the entropy regularization term, an attenuation coefficient that attenuates degree to which the portion corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood as the process of updating the parameter is repeated, and update the parameter of the reward function to maximize the set lower limit of log-likelihood2.”
This limitation does nothing to alter the fundamental nature of the claims as a mathematical concept. Under its BRI, in light of the specification, this limitation encompasses the mathematical concept of setting/establishing a coefficient that attenuates a degree to which a portion of data values corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood and updating the parameter of the reward function (i.e., a mathematical function) in order to maximize the lower limit of log-likelihood (See, e.g., paragraphs 48, 50-54, 56 and 85-87).
Step 2A Prong 2 Analysis: Mere instructions to apply the mathematical concept electronically do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The claims do not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claims are subject-matter ineligible.
For example, claims 2, 8 and 11 only recite the additional elements of “wherein the processor is configured to execute the instructions to” <perform the above-noted setting operation> (claim 2), “wherein the computer” <performs the above-noted set step> (claim 8) and “The non-transitory computer readable information recording medium” (claim 10), which are mere instructions to apply the mathematical concept. Mere instructions to apply the mathematical concept electronically (i.e., with “the processor” of claims 2 and 11, and “the computer” of claim 8), do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Also, the step of “maximizing the lower limit of the log-likelihood as the process of updating the parameter is repeated” is performing repetitive calculations and can be characterized as insignificant extra solution activity. See MPEP 2106.05(g).
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “The learning device according to claim 1, wherein the processor is configured to execute the instructions to” <perform the above-noted set operation> recited in claim 2, “wherein the computer” <performs the above-noted set step> of claim 8 and “The non-transitory computer readable information recording medium” of claim 11) do not amount to significantly more than the judicial exception.
Also, “maximizing the lower limit of the log-likelihood as the process of updating the parameter is repeated” can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
Regarding claims 3, 9 and 12, these claims are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a device as depending from claim 1, claim 9 is directed to a method as depending from claim 7, and claim 12 is directed to a computer readable information recording medium as depending from claim 10, thus the analysis for patent eligibility of claims 1, 7 and 10, respectively, are incorporated herein.
Step 2A Prong 1: The claims recite, using respective similar language, “set an
attenuation coefficient that attenuates degree to which the entropy regularization term contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term, and, in the course of repeating the process of updating the parameter, change the attenuation coefficient to attenuate degree to which the portion corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood3.”
This limitation does nothing to alter the fundamental nature of the claims as a mathematical concept. Under its BRI, in light of the specification, this limitation encompasses the mathematical concept of setting/establishing a coefficient that attenuates a degree to which a portion of data values corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term in order to maximize the lower limit of log-likelihood (See, e.g., paragraphs 48, 50-54, 56 and 85-87).
Step 2A Prong 2 Analysis: Mere instructions to apply the mathematical concept electronically do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The claims do not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claims are subject-matter ineligible.
For example, claims 3, 9 and 12 only recite the additional elements of “wherein the processor is configured to execute the instructions to” <perform the above-noted setting operation> (claim 3), “wherein the computer” <performs the above-noted set step> (claim 9) and “The non-transitory computer readable information recording medium” (claim 12), which are mere instructions to apply the mathematical concept. Mere instructions to apply the mathematical concept electronically (i.e., with “the processor” of claims 3 and 12, and “the computer” of claim 9), do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Also, the step of “in the course of repeating the process of updating the parameter, change the attenuation coefficient” is performing repetitive calculations and can be characterized as insignificant extra solution activity. See MPEP 2106.05(g).
Step 2B Analysis: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “The learning device according to claim 1, wherein the processor is configured to execute the instructions to” <perform the above-noted set operation> recited in claim 3, “wherein the computer” <performs the above-noted set step> of claim 9 and “The non-transitory computer readable information recording medium” of claim 12) do not amount to significantly more than the judicial exception.
Also, “in the course of repeating the process of updating the parameter, change the attenuation coefficient” can be characterized as insignificant extra solution activity that is well understood routine and conventional. See MPEP 2106.05(g) and MPEP 2106.05(d)(II) example (ii) provides that performing repetitive calculations has been understood by the courts to be well-understood, routine and conventional.
Regarding claim 4, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a device as depending from claim 3, thus the analysis for patent eligibilities of claim 3 and of base claim 1 are incorporated herein.
Step 2A Prong 1: The claim recites “change the attenuation coefficient when it is determined that the moving average of the log-likelihood has become constant4.”
This limitation does nothing to alter the fundamental nature of the claim as a mathematical concept. Under its BRI, in light of the specification, this limitation encompasses the mathematical concept of changing/modifying the attenuation coefficient responsive to determining that a moving average of the log-likelihood has become constant/static (See, e.g., paragraphs 55 and 87).
Step 2A Prong 2 Analysis: Mere instructions to apply the mathematical concept electronically do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
For example, the claim only recites the additional element of “wherein the processor is configured to execute the instructions to” <perform the above-noted change operation>, which are mere instructions to apply the mathematical concept. Mere instructions to apply the mathematical concept electronically (i.e., with “the processor”) do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “The learning device according to claim 3, wherein the processor is configured to execute the instructions to” <perform the above-noted change operation> recited in claim 4 does not amount to significantly more than the judicial exception.
Regarding claim 5, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a device as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim recites “derive the lower bound for the log-likelihood based on an upper bound of a log sum exponential5.”
This limitation does nothing to alter the fundamental nature of the claim as a mathematical concept. Under its BRI, in light of the specification, this limitation encompasses the mathematical concept of deriving the lower bound or limit value for the log-likelihood based on an upper bound or limit value of a log sum exponential (i.e., a mathematical calculation) (See, e.g., paragraphs 38, 88 and 99).
Step 2A Prong 2 Analysis: Mere instructions to apply the mathematical concept electronically do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
For example, the claim only recites the additional element of “wherein the processor is configured to execute the instructions to” <perform the above-noted derive operation>, which are mere instructions to apply the mathematical concept. Mere instructions to apply the mathematical concept electronically (i.e., with “the processor”) do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “The learning device according to claim 1, wherein the processor is configured to execute the instructions to” <perform the above-noted derive operation> recited in claim 5 does not amount to significantly more than the judicial exception.
Regarding claim 6, this claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a device as depending from claim 1, thus the analysis for patent eligibility of claim 1 is incorporated herein.
Step 2A Prong 1: The claim does not recite any judicial exception. However, it is still directed to the same abstract idea as identified in claim 1, discussed above.
Step 2A Prong 2 Analysis:
The claim recites the additional limitations: “wherein the processor is configured to execute the instructions to accept input of the reward function whose feature is set to be a linear function.”
As drafted, the limitation “accept input of the reward function whose feature is set to be a linear function” amounts to an insignificant extra-solution activity (i.e., accepting input data is mere data gathering), which does not integrate a judicial exception into a practical application. See MPEP 2106.05(g). That is, the additional element of “accept input of the reward function whose feature is set to be a linear function” is adding insignificant extra-solution activity (amounts to necessary data gathering) to the judicial exception, as discussed in MPEP § 2106.05(g).
Also, mere instructions to apply the mathematical concept electronically (i.e., “wherein the processor is configured to execute the instructions to” <perform the above-noted accept operation> do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
For example, the claim only recites the additional element of “wherein the processor is configured to execute the instructions to” <perform the above-noted derive operation>, which are mere instructions to apply the mathematical concept. Mere instructions to apply the mathematical concept electronically (i.e., with “the processor”) do not meaningfully integrate the judicial exception into a practical application. See MPEP 2106.05(f).
Therefore, the claim does not recite any additional elements that integrate the abstract idea into a practical application or provide significantly more than the abstract idea, and thus the claim is subject-matter ineligible.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Mere instructions to apply the mathematical concept electronically (i.e., with the recited “The learning device according to claim 1, wherein the processor is configured to execute the instructions to” <perform the above-noted accept operation> recited in claim 6 does not amount to significantly more than the judicial exception.
As discussed above with respect to integration of the abstract idea into a practical application, all of the additional elements are insignificant extra-solution activities or mere instructions to apply an exception. Insignificant extra-solution activities and mere instructions to apply an exception cannot provide an inventive concept.
Moreover, receiving, communicating, and storing data are insignificant extra-solution activities that are well-understood, routine, and conventional. See MPEP2106.05(d)(II) (“The courts have recognized the following computer functions as well‐understood, routine, and conventional functions… i. Receiving or transmitting data over a network…iv. Storing and retrieving information in memory”) (citing OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). Therefore, the recitation of “accept input of the reward function whose feature is set to be a linear function” are the well-understood, routine, conventional activities of receiving or transmitting data over a network, as discussed in MPEP § 2106.05(d).
Conclusion
The claims have been searched insofar as they can be understood. The prior art made of record, listed on form PTO-892, and not relied upon, is considered pertinent to applicant's disclosure; and all references generally relate to the implementation and use of reward functions and reinforcement learning.
For example, Yoon et al. (U.S. Patent Pub. No. 2022/0405682 A1, hereinafter “Yoon”) discloses “an inverse reinforcement learning-based delivery means detection apparatus and method” using “a reinforcement learning policy agent configured using an artificial neural network, and an inverse reinforcement learning reward network (i.e., a reward function) configured using an artificial neural network modeling a distributional difference between an action pattern imitated by the policy agent and an actual action pattern a reinforcement learning policy agent configured using an artificial neural network, and an inverse reinforcement learning reward network (i.e., a reward function) configured using an artificial neural network modeling a distributional difference between an action pattern imitated by the policy agent and an actual action pattern” where “The maximum entropy IRL models expert demonstration using a Boltzmann distribution, and the reward function is modeled as a parameterized energy function of the trajectories” and “This framework assumes that the expert trajectory is close to an optimal trajectory with the highest likelihood. In this model, optimal trajectories defined in a partition function Z are exponentially preferred. Since determining the partition function is a computationally difficult challenge, early studies in the maximum entropy IRL suggested dynamic programming in order to compute Z.” (see, Abstract, and paragraphs 54 and 62-63).
Yoon also discloses “The log-likelihood term inside the expectation is necessarily the same as applying the logarithm to the likelihood defined in Formula 2. Accordingly, estimating the expectation term also fulfills the need for z estimation. Unlike the previous approaches that estimate Z within the likelihood term using backup trajectory samples together with MCMC [Markov chain Monte Carlo], the present invention uses the learned parameters to measure the difference in posterior distribution between expert rewards and policy rewards. Then, the log-likelihood term may be approximated using a marginal Gaussian log-likelihood (GLL). Since a plurality of parameters may be used when a plurality of features of the posterior are assumed, the present invention may use the mean of a plurality of GLL values.” (see, paragraphs 68 and 77).
Also, for example, non-patent literature Zhang et al. (“Learning invariant representations for reinforcement learning without reconstruction." arXiv preprint arXiv:2006.10742 v2 (7 April 2021), hereinafter “Zhang”) discloses “how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.”, “We show this using the property that the optimal value function is Lipschitz with respect to the bisimulation metric”, “the learned representation will generalize to unseen reward functions, as long as the new reward function has a subset of the same causal ancestors. As an example, a representation learned for a robot to walk will likely generalize to learning to run, because the reward function depends on forward velocity and all the factors that contribute to forward velocity. However, that representation will not generalize to picking up objects, as those objects will be ignored by the learned representation, since they are not likely to be causal ancestors of a reward function designed for walking.” and “The policy improvement step then attempts to project a parametric policy π(at|st) by minimizing KL divergence between the policy and a Boltzmann distribution” (see, Abstract and pages 5-6 and 16).
The examiner requests, in response to this office action, support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line no(s) in the specification and/or drawing figure(s). This will assist the examiner in prosecuting the application.
When responding to this office action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the reference cited or the objections made. He or she must also show how the amendments avoid such references or objections See 37 CFR 1.111 (c).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RANDY K BALDWIN whose telephone number is (571)270-5222. The examiner can normally be reached on Mon - Fri 9:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached on 571-272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RANDALL K. BALDWIN/Primary Examiner, Art Unit 2125
1 As indicated above in the section 112(b) rejections of these claims, “a trajectory that minimizes Wasserstein distance, which represents distance between a probability distribution of a trajectory of an expert and a probability distribution of a trajectory determined based on a parameter of the reward function” has been interpreted to be any trajectory, history, trend or path that minimizes a Wasserstein distance between a probability distribution of a trajectory, direction, history, trend or path of decision-making data of an expert, such as, but not limited to state and action pairs, and a probability distribution of any trajectory, direction, history, trend or path determined based on a parameter of the reward function.
2 As indicated above in the section 112(b) rejections of these claims, “an attenuation coefficient that attenuates degree to which the portion corresponding to the entropy regularization term contributes to maximizing the lower limit of the log-likelihood” has been interpreted to be any coefficient that attenuates a degree to which any portion or subset of data corresponding to or associated or correlated with the previously-introduced entropy regularization term that contributes to maximizing the lower limit of the log-likelihood similarity.
3 As indicated above in the section 112(b) rejections of these claims, “an attenuation coefficient that attenuates degree to which the entropy regularization term contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term” has been interpreted to be any coefficient that attenuates a degree to which any portion or subset of data corresponding to or associated or correlated with the previously-introduced entropy regularization term that contributes to maximizing the lower limit of the log-likelihood to the portion corresponding to the entropy regularization term
4 As indicated above in the section 112(b) rejection of this claim, “the moving average of the log-likelihood” has been interpreted as any moving average of the log-likelihood.
5 As indicated above in the section 112(b) rejections of this claim, “the lower bound for the log-likelihood” has been interpreted as any lower bound value or lower limit value for the previously-introduced log-likelihood, and “an upper bound of a log sum exponential” has been interpreted as any upper bound value, maximum value, or highest limit value, cutoff or threshold of a log sum exponential
Prosecution Insights