Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The action is in response to claims dated 3/30/2023
Claims pending in the case: 1-20
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-6, 8-10, 13-14, 16-17, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Prashanth (Risk-Sensitive Reinforcement Learning via Policy Gradient Search) in view of Ostrovski (US 20230196108).
Regarding Claim 1, Prashanth teaches, A computer-implemented method for modifying a current policy using reinforcement learning (RL) (Prashanth: Pg. 1 abstract), comprising:
sampling a number, corresponding to an inputted sample size, of Markov Decision Processes (MDPs) defining an environment (Prashanth: Section 1, Pg. 3 [1], Pg. 15 section 2: MDP framework; Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm data sampling);
for each of the sampled MDPs, collecting behavior data for the current policy (Prashanth: Pg. 7 [2], Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm -using policy sample system),
determining a quantile function of return with the current policy using the collected behavior data, and generating a current weight by updating a weight for a particular sampled MDP using the quantile function of return for the particular sampled MDP (Prashanth: Pg. 5 [3], Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm – estimate functions; Pg. 32-33 section 3.4, Pg. 93 section 6.1: Value at risk function is a quantile function); and
modifying the policy based upon the current weight for each of the sampled MDPs (Prashanth: Pg. 7 [2], Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm -using policy sample system; Pg. 139 Algorithm 7: policy update);
Although the data collection taught in Prashanth reads on the limitation as claimed, Prashanth does not specifically recite behavior data,
Ostrovski teaches,
behavior data (Ostrovski: Fig. 5, [7, 11, 16, 57-58, 77]: use quantum function to update network parameters based on observation on current polity);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Prashanth and Ostrovski because the combination would enable generating quantile function to update policy. One of ordinary skill in the art would have been motivated to combine the teachings because the combination uses agent behavior data to update policy as is common in the art.
Regarding claim 2, Prashanth and Ostrovski teach the invention as claimed in claim 1 above and, wherein the current weights are generated by minimizing a conditional value of at risk (CVaR) of a return of the current policy, and the policy is modified to maximize a weighted average of the CVaR of the return with the current weights (Pg. 32-33 section 3.4, Pg. 109 section 6.3, Pg. 135-138 section 7.3: constrained optimization).
Regarding claim 5, Prashanth and Ostrovski teach the invention as claimed in claim 2 above and, wherein the sampled MDPs are sampled based upon a prior distribution of the environment (Prashanth: Section 1, Pg. 3 [1], Pg. 15 section 2: MDP framework; Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm data sampling) (Ostrovski: [58-59]: sampling engine);
Regarding claim 6, Prashanth and Ostrovski teach the invention as claimed in claim 2 above and, wherein the weight is updated by solving an optimization problem (Prashanth: Pg. 7 [2], Pg. 8 Figure 1.1: risk sensitive policy gradient algorithm -using policy sample system; Pg. 139 Algorithm 7: optimize parameters) (Ostrovski: Fig. 5, [7, 11, 16, 77]: optimize).
Regarding claim 8, Prashanth and Ostrovski teach the invention as claimed in claim 2 above and, wherein the behavior data is determined for the current policy by interacting with the environment using the current policy (Ostrovski: [11, 58-59]: current interaction data).
Regarding Claim(s) 9-10, 13-14, 16, this/these claim(s) is/are similar in scope as claim(s) 1-2, 5-6, 8 respectively. Therefore, this/these claim(s) is/are rejected under the same rationale.
Regarding Claim(s) 17, this/these claim(s) is/are similar in scope as claim(s) 1 respectively. Therefore, this/these claim(s) is/are rejected under the same rationale.
Regarding claim 19, Prashanth and Ostrovski teach the invention as claimed in claim 17 above and, wherein the weight is updated by solving an inner minimization problem (Ostrovski: [46, 68, 83]: updating network parameters).
Claim(s) 3, 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Prashanth (Risk-Sensitive Reinforcement Learning via Policy Gradient Search) and Ostrovski (US 20230196108) in view of Fairbank (US 10417556).
Regarding claim 3, Prashanth and Ostrovski teach the invention as claimed in claim 2 above but not, wherein a variational auto-encoder (VAE) including an encoder and a decoder is optimized using the collected behavior data;
Fairbank teaches, wherein a variational auto-encoder (VAE) including an encoder and a decoder is optimized using the collected input data (Fairbank: col 10 lines 46-68, col 30 lines 58-67, col 31 lines 13-25: a VAE may be used on the input data);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Prashanth, Ostrovski and Fairbank because the combination would enable using a VAE. One of ordinary skill in the art would have been motivated to combine the teachings because the combination provides an additional benefit of improved accuracy (see Fairbank col 30 lines 42-60).
Regarding Claim(s) 11, this/these claim(s) is/are similar in scope as claim(s) 3 respectively. Therefore, this/these claim(s) is/are rejected under the same rationale.
Claim(s) 4, 12, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Prashanth (Risk-Sensitive Reinforcement Learning via Policy Gradient Search) and Ostrovski (US 20230196108) and Fairbank (US 10417556) in view of Ozay (US 20230145919).
Regarding claim 4, Prashanth, Ostrovski and Fairbank teach the invention as claimed in claim 3 but not, wherein the VAE is optimized by maximizing an evidence lower bound (ELBO);
Ozay teaches, wherein the VAE is optimized by maximizing an evidence lower bound (ELBO) (Ozay: [156]: training may maximize ELBO);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Prashanth, Ostrovski and Fairbank and Ozay because the combination would enable optimizing a VAE by maximizing ELOB as is often practiced in the art.
Regarding Claim(s) 12, this/these claim(s) is/are similar in scope as claim(s) 4 respectively. Therefore, this/these claim(s) is/are rejected under the same rationale.
Regarding Claim(s) 18, this/these claim(s) is/are similar in scope as claim(s) 11 and 12. Therefore, this/these claim(s) is/are rejected under the same rationale.
Claim(s) 7, 15, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Prashanth (Risk-Sensitive Reinforcement Learning via Policy Gradient Search) and Ostrovski (US 20230196108) in view of Choi (US 20220198225).
Regarding claim 7, Prashanth and Ostrovski teach the invention as claimed in claim 2 above but not, wherein the quantile function of return is determined using distributional RL;
Choi teaches, wherein the function of return is determined using distributional RL (Choi: [105, 121]: issues found in RL may be overcome by using distribution RL);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Prashanth, Ostrovski and Choi because the combination would enable using distributional RL. The combination provides an improvement such as superior performance and sample efficiency as is known in the art (see Choi 121-122).
Regarding Claim(s) 15 and 20, this/these claim(s) is/are similar in scope as claim(s) 7. Therefore, this/these claim(s) is/are rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure in attached 892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MANDRITA BRAHMACHARI whose telephone number is (571)272-9735. The examiner can normally be reached Monday to Friday, 11 am to 8 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571 272 4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Mandrita Brahmachari/Primary Examiner, Art Unit 2144
Prosecution Insights