DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 are pending for examination.
Claims 1, 8 and 15 are independent Claims.
Claims 1-20 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Frazier et al. (U.S. 2004/0064735 hereinafter Frazier) in view of Oroojloojadid (U.S. 2022/0374732 hereinafter Jadid) in further view of Reeb et al. (U.S. 2021/0187734 hereinafter Reeb).
As Claim 1, Frazier teaches a computer-implemented method comprising:
receiving a policy, the policy for acting in an environment having a set of states (Frazier (¶0042 last 3 lines), a responsive policy is generated from statistical estimation);
responsive to determining that the policy is a non-deterministic policy (Frazier (¶0144 line 1-4, ¶0148 line 4-7, fig. 6), control is assigned to WN. Selected controller in figure 6 is generated under a “uncontrolled” policy; thus, actuation of a selected active control can have no effect),
responsive to determining that a state-transition function associated with the set of states is unknown, approximating the state-transition function (Frazier (¶0042 line 7-13, ¶0085 line 1-5), system detect imperfect state information. Statistical estimation predicts the current state estimation. System includes transition probability matrix, transition cost and observation probability matrices); and
verifying using a constraint solver the policy with the state-transition function (Frazier (¶0042 last 3 lines, ¶0095 line 1-7), statistical estimation predicts response policy based on state estimation. Control objective is to minimize the overall average cost per state).
Frazier may not explicitly disclose:
generated by deep reinforced learning
at least by training a deep neural network and transforming the deep neural network into a polynomial
Jadid teaches:
generated by deep reinforced learning (Jadid (¶0039 last 6 lines, ¶0041 last 3 lines), models are trained using deep neural network and reinforcement learning)
at least by training a deep neural network and transforming the deep neural network into a polynomial (Jadid (¶0039 last 6 lines, ¶0041 last 3 lines, ¶0046 line 11-15), model are trained using deep neural network and reinforcement learning. Machine learning type can be non-linear regression via a Taylor series expansion)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify machine model of Frazier instead be a deep neural network reinforcement learning machine taught by Jadid, with a reasonable expectation of success. The motivation would be to “multi-task learning has been extended to deep-learning
methods to produce deep neural networks that are capable of solving for multiple tasks simultaneously” (Jadid (¶0041 last 3 lines)).
Frazier in view of Jadid may not explicitly disclose:
decomposing the non-deterministic policy into a set of deterministic policies,
wherein non-deterministic policy includes a set of action from which to choose for taking;
based on the verifying, shielding a deep reinforced learning agent from failing into an unsafe state in a real world environment.
Reeb teaches:
decomposing the non-deterministic policy into a set of deterministic policies (Reeb (¶0150 line 1-7), “To update the safe set, method 700 may comprise, in an operation titled "ESTIMATING TRANSITION PROBABILITY BASED ON OTHER PAIR", estimating 732 a transition probability for a state-action pair (non-deterministic policy includes all state-action pairs) based on an empirical transition probability of a similar other state action pair. To update the safe set (deterministic policy is the safe set of actions), method 700 may comprise, in an operation titled "INCLUDING PAIR IF SAFE",”), wherein non-deterministic policy includes a set of action from which to choose for taking (Reeb (¶0149 line 1-6), “the method 700 may comprise, in an operation titled "OBTAINING CURRENT STATE", obtaining 720 data indicating a current state of the physical environment. In the iteration 720-750, the method 700 may comprise, in an operation titled "UPDATING SAFE SET", updating 730 the safe set of state-action pairs”);
based on the verifying (Rebb (¶0034 line 10-16), “For example, a state-action pair with smallest confidence intervals, or smallest upper end of the confidence interval, may be selected. The estimated transition probabilities of that state-action pair may then be translated to the present state-action pair. This way, more reliable estimates may be obtained, decreasing the probability of inadvertently performing an unsafe action.”), shielding a deep reinforced learning agent from failing into an unsafe state in a real-world environment (Reeb (¶0151 line 2-5), “"SELECTING ACTION FROM SAFE SET", selecting 740 an action to be performed in the current state of the physical environment from the safe set of state-action pairs”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify policies of Frazier in view of Jadid instead be policies for robot state-action pair taught by Reeb, with a reasonable expectation of success. The motivation would be to provide “more reliable estimates may be obtained, decreasing the probability of inadvertently performing an unsafe action” (Reeb (¶0034 line 10-16)).
As Claim 2, besides Claim 1, Frazier in view of Jadid in further view of Reeb teaches wherein the decomposing of the policy into the set of deterministic policies includes:
dividing a probability distribution associated with the non-deterministic policy into a plurality of regions (Frazier (¶0133), observation space Z is divided into KER, PID and IP regions);
from each of the plurality of regions, selecting a sample, wherein the set of deterministic policies includes the sample from each of the plurality of regions (Frazier (¶0133), Z is determined from samples from each region).
As Claim 3, besides Claim 2, Frazier in view of Jadid in further view of Reeb teaches wherein the sample represents a mean of a region from which the sample is selected (Frazier (¶0133), sample represents the value for the region).
As Claim 4, besides Claim 2, Frazier in view of Jadid in further view of Reeb teaches wherein the sample represents a mode of a region from which the sample is selected (Frazier (¶0133), sample represents the binary mode of the region).
As Claim 5, besides Claim 1, Frazier in view of Jadid in further view of Reeb teaches wherein the approximating the state-transition function includes:
collecting a set of trajectories by running the policy in the environment, the set of trajectories including a sequence of state and action at each time step (Frazier (¶0015 line 4-14), controller includes two or more states based on one or more objectives of an electronic attack. One or more state transition probabilities based on a state at a state k and a control at a state k, a control at state k-1 …);
training the deep neural network based on the set of trajectories to predict a next state at next time step (Frazier (¶0015 line 4-14, ¶0124 line 2-4), recursive estimator generates a probabilistic state estimate for stage K and a response policy for state K, PO-MDP model is trained on a training set); and
transforming parameters of the deep neural network into a polynomial function using a I- th order Taylor approximation (Jadid (¶0039 last 6 lines, ¶0041 last 3 lines, ¶0046 line 11-15), model are trained using deep neural network and reinforcement learning. Machine learning type can be non-linear regression via a Taylor series expansion), wherein the polynomial function used during the verifying as the state-transition function (Jadid (¶0239 last 5 lines, ¶0230 line 13-15), model is verified through testing. System predicts next state from current state).
As Claim 6, besides Claim 1, Frazier in view of Jadid in further view of Reeb teaches wherein the shielding including performing runtime shielding (Rebb (¶0034 line 10-16), “more reliable estimates may be obtained, decreasing the probability of inadvertently performing an unsafe action.”).
As Claim 7, besides Claim 6, Frazier in view of Jadid in further view of Reeb teaches wherein the runtime shielding includes, at a time step during runtime:
generating an action for current state using the policy (Frazier (¶0144 line 1-4, ¶0148 line 4-7, fig. 6), control is assigned to WN. Selected controller in figure 6 is generated under a “uncontrolled” policy; thus, actuation of a selected active control can have no effect);
identifying from the set of deterministic policies, a deterministic policy most likely to produce the generated action;
predicting a next state that would result from the action given the state-transition function (Frazier (¶0148 line 7-16, fig. 7(a), ¶0149, ¶0150, fig. 7(b)), under heuristic policy, control K and WA are selected);
checking an inductive invariant of the identified deterministic policy with the predicted next state (Frazier (¶0147 last 10 lines, fig. 6), plurality of values are used to evaluate the effectiveness of current control);
As Claims 8-14, the Claims are rejected for the same reasons as Claims 1-7, respectively.
As Claims 15-12, the Claims are rejected for the same reasons as Claims 1-5 and 7, respectively.
Response to Arguments
Claim Rejections – 35 U.S.C. §101:
Applicants’ arguments are persuasive; therefore, 35 U.S.C. §101 rejection(s) on the Claims are respectfully withdrawn.
Claim Rejections – 35 U.S.C. §103:
As Claim 1-20, Applicants argue that Frazier does not disclose “decomposing a non-deterministic policy into a set of deterministic policy wherein …” (first paragraph of page 12 in the remarks).
PNG
media_image1.png
108
653
media_image1.png
Greyscale
Applicants’ arguments are moot because new reference Reeb teaches the limitation(s).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/ Primary Examiner, Art Unit 2147
Prosecution Insights