Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The Examiner notes the applicant’s assertion that: “A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se”, as stated in paragraph 36 of the filed specification.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “a best action” in Claim 1 is a relative term which renders the claim indefinite. The term “a best action” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The same rejection is made for independent Claims 7 and 13. Dependent claims are subsequently rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1:
All claims are directed towards either a method, a product or a system and thus satisfies Step 1 as falling into one of the statutory categories.
Step 2A, Prong One:
Independent Claim 1 recites (the same analysis applies to similar independent Claims 7 and 13):
computing utilities for respective candidate actions at a current time step, using a return distribution predictor;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing with pen and paper, objective values or utilities, from a return distribution function or predictor.
computing a utility gap between a utility of a best action at the current time step and a utility of a reference action;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing a difference or gap between two action functions using evaluation.
computing a threshold at the current time step for the utility gap;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of computing a threshold value pertaining to the above difference/gap using evaluation.
determining whether the utility gap is greater than the threshold;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of determining whether the difference/gap is greater than the threshold using evaluation.
in responding to determining the utility gap being greater than the threshold, accepting the best action at the current time step;
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of making a determination to accept the best action based difference/gap being greater than the threshold, using judgement.
and in response to determining the utility gap being not greater than the threshold, at the current time step, rejecting the best action
this limitation, under its broadest reasonable interpretation, covers concepts that can be performed in the human mind and therefore would fall under the “Mental Processes” groupings of abstract ideas. That is the human mind is capable of making a determination to reject the best action based difference/gap not being greater than the threshold, using judgement.
Step 2A, Prong Two:
Claim 1 recites the additional elements of (the same analysis applies to similar independent Claims 7 and 13):
adaptively-repeated action selection in reinforcement learning,
this is considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
and repeating an action that has been taken at a previous time step.
This is considered as adding insignificant extra-solution activity (iterative processing) to the judicial exception - see MPEP 2106.05(g).
The further additional elements of “computer” and/or “processors” as recited in independent claims 7 and 13 are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are therefore directed to an abstract idea.
Step 2B:
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements are considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h), and as well understood, routine conventional activity as identified by the court similar to performing repetitive calculations - see MPEP 2106.05(d). The further additional elements of “computer” and/or “processors” as recited in these independent claims amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claims are therefore not patent eligible.
Dependent Claims 2-6, and similar Claims 8-12, and 14-18 are considered as generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
The closest prior art of Ostrovski, US 2020/0364557 A1, teaches distributional reinforcement learning and return distribution, however, the prior art does not appear to teach all of the interrelated limitations of the independent claims, singularly or in combination, as stated and as claimed; and as specifically pointed out in independent Claim 1 (and similarly in independent Claims 7 and 13), “computing a utility gap between a utility of a best action at the current time step and a utility of a reference action; computing a threshold at the current time step for the utility gap; determining whether the utility gap is greater than the threshold; in responding to determining the utility gap being greater than the threshold, accepting the best action at the current time step; and in response to determining the utility gap being not greater than the threshold, at the current time step, rejecting the best action and repeating an action that has been taken at a previous time step”.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See PTO-892 for the relevant prior art where for example the NPL of Duan teaches a distribution function/predictor in reinforcement learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVE MISIR whose telephone number is (571)272-5243. The examiner can normally be reached M-R 8-5 pm, F some hours.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at 5712703169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVE MISIR/Primary Examiner, Art Unit 2127