Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Applicant is required to explain how each and every reference listed on the 51 page IDS is related to the present claimed invention. The information disclosure statement filed 1/30/23 fails to comply with 37 CFR 1.98 (a) (3) (i) because it does not include a concise explanation of the relevance, as it is presently understood by the individual designated in 37 CFR 1.56 (c) most knowledgeable about the content of the information, of each reference listed that is not in the English language. It has been placed in the application file, but the information referred to therein has not been considered. There is clearly no need for a 51 page IDS, especially without clear concise explanation of the relevance of every reference. It is not advisable to bury the most relevant reference(s) in a sea of references. Applicant should go through every single cited reference and indicate how they pertain to the present claimed invention.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: claims 1-17 are directed to either a process, machine, manufacture or composition of matter.
With respect to claim 1:
2A Prong 1:
It is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016); see SAP Am., Inc. v. InvestPic, LLC, 898 F.3d 1161, 1167 (Fed. Cir. 2018) ("[S]electing certain information, analyzing it using mathematical techniques, and reporting or displaying the results of the analysis" is abstract); Intellectual Ventures I LLC v. Cap. One Fin. Corp., 850 F.3d 1332, 1341 (Fed. Cir. 2017) ("Organizing, displaying, and manipulating data of particular documents" is abstract.); FairWarning IP, LLC v. Iatric Sys., Inc., 839 F.3d 1089, 1096-97 (Fed. Cir. 2016) (compiling and combining disparate data sources to generate a full picture of a user's activity, identity, frequency of activity, and the like in a computer environment to detect potential fraud does not differentiate a process from ordinary mental processes); In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022) ("These steps can be performed by a human, using 'observation, evaluation, judgment, [and] opinion,' because they involve making determinations and identifications, which are mental tasks humans routinely do").
iteratively improve a behavior policy of the artificial agent through a plurality of behavior policy improvement iterations (Abstract idea of analyzing data. Mental process. A human- mind with pen and paper can generate/determine data);
the improved behavior policy in each behavior policy improvement iteration being selected from a plurality of behavior policies in a space of stochastic behavior policies (Abstract idea of analyzing data. Mental process. A human can select data/policies);
a diverse exploration strategy is employed which strives for behavior diversity in the space of stochastic policies, by deploying a diverse set comprising a plurality of behavior policies which are ensured as being safe while excluding inferior behavior policies which do not exceed a lower bound of behavior policy performance determined based on prior assessment of behavior policies (encompasses mental observations or evaluations, e.g., a computer programmer’s mental identification of data);
behavior policy performance of each of the plurality of behavior policies policy of the diverse set is assessed with respect to operation of the artificial agent for each iteration of the iterative improvement(encompasses mental observations or evaluations, e.g., a computer programmer’s mental identification of data);
the lower bound of behavior policy performance is updated based on the assessed behavior policy performance(mental process of modeling and/or performing mathematical operations with assistance of pen and paper).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A system for learning, automated processor, artificial agent (computer component is recited at a high level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component; the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention." Alice, 134 S. Ct. at 2358);
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A system for learning, automated processor, artificial agent (computer component is recited at a high level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component; the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention." Alice, 134 S. Ct. at 2358).
The claim is not patent eligible.
2. (Currently Amended) The system according to claim 1, wherein each behavior policy has a variance associated with an estimate of its behavior policy performance by importance sampling, and each diverse set has a common average variance, in each of a plurality of behavior policy improvement iterations (further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
3. (Currently Amended) The system according to claim 1, wherein each of the behavior policy performance and behavior policy behavior diversity is quantified according to a common objective function, and the at least one automated processor is further configured to employ the common objective function to assess the behavior policy performance of the diverse set(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
4. (Currently Amended) The system according to claim 1, wherein the diverse set comprises a plurality of behavior policies predefined upon commencement of a respective single iteration of behavior policy improvement iteration of the plurality of behavior policy improvement iterations, the system further comprising at least one memory configured to store the updated lower bound of behavior policy performance(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
5. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to adaptively define the diverse set based on assessed behavior policy performance within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
6. (Currently Amended) The system according to claim 5, wherein the at least one automated processor is further configured to control the adaptation based on at least a change in the lower bound of behavior policy performance as a selection criterion for a subsequent diverse set within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
7. (Currently Amended) The method system according to claim 5, wherein the at least one automated processor is further configured to control the adaptation based on at least feedback of a system state of the artificial agent received after deploying a prior behavior policy within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
8. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to select the diverse set within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations as the plurality of behavior policies generated based on prior feedback, having maximum differences from each other according to a Kullback-Leibler (KL) divergence measure(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
9. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to select the plurality of behavior policies of the diverse set within a respective iteration of the plurality of behavior policy improvement iterations according to an aggregate group statistic(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
10. (Currently Amended) The system according claim 1 further comprising the artificial agent controlled according to a respective behavior policy from the iteratively improved diverse set,
wherein the at least one automated processor is further configured to, during each behavior policy improvement iteration of the plurality of behavior policy improvement iterations: select the diverse set of behavior policies from a space of stochastic behavior policies according to a selection criterion; ensure that each respective behavior policy is safe and has a statistically expected return no worse than the lower threshold bound of behavior policy performance; maximize behavior diversity of the diverse set according to a diversity metric; updates update the selection criterion; and store the selection criterion in a memory (further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016, memory is a computer component is recited at a high level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component; the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention." Alice, 134 S. Ct. at 2358);
11. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to employ importance sampling within a confidence interval to select the plurality of behavior policies within the diverse set for each behavior policy improvement iteration of the plurality of policy improvement iterations(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
12. (Currently Amended) The system according to claim 1, further comprising: an input port configured to receive a data set representing an environment in a first number of dimensions in each behavior policy improvement iteration of the plurality of behavior policy improvement iterations, wherein the at least one automated processor is further configured to collect the data set representing an environment in a first number of dimensions in each iteration, and wherein the diverse set of behavior policies have a second number of dimensions less than the first number of dimensions(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
13. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to update a statistically expected return, no worse than [[a]] the lower bound of behavior policy performance, between the plurality of behavior policy improvement iterations(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
14. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured, in each behavior policy improvement iteration, to obtain feedback data from a system controlled by the artificial agent in accordance with the respective behavior policy, and to use the feedback data to improve a computational model of the system which is predictive of future behavior of the system over a range of environmental conditions(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
15. (Currently Amended) The system according to claim 14, wherein the at least one automated processor is further configured to implement the computational model of the system which is predictive of future behavior of the system over a multidimensional range of environmental conditions, based on a plurality of observations under different environmental conditions having a distribution, and to bias the diverse exploration strategy to select respective behavior policies within the diverse set of behavior policies which selectively explore portions of the multidimensional range of environmental conditions(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
16. (Original) The system according to claim 1, wherein the at least one automated processor is further configured to select the diverse set based on a predicted state of a system controlled by the artificial agent according to the respective behavior policy during deployment of the respective behavior policy(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
17. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to select the diverse set for assessment within each behavior policy improvement iteration of the plurality of behavior policy improvement iterations to generate a maximum predicted statistical improvement in behavior policy performance(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
Claims 18-20. (Cancelled)
Claims 21-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: claims 21-23 are directed to either a process, machine, manufacture or composition of matter.
With respect to claims 21,22:
2A Prong 1:
It is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016); see SAP Am., Inc. v. InvestPic, LLC, 898 F.3d 1161, 1167 (Fed. Cir. 2018) ("[S]electing certain information, analyzing it using mathematical techniques, and reporting or displaying the results of the analysis" is abstract); Intellectual Ventures I LLC v. Cap. One Fin. Corp., 850 F.3d 1332, 1341 (Fed. Cir. 2017) ("Organizing, displaying, and manipulating data of particular documents" is abstract.); FairWarning IP, LLC v. Iatric Sys., Inc., 839 F.3d 1089, 1096-97 (Fed. Cir. 2016) (compiling and combining disparate data sources to generate a full picture of a user's activity, identity, frequency of activity, and the like in a computer environment to detect potential fraud does not differentiate a process from ordinary mental processes); In re Killian, 45 F.4th 1373, 1379 (Fed. Cir. 2022) ("These steps can be performed by a human, using 'observation, evaluation, judgment, [and] opinion,' because they involve making determinations and identifications, which are mental tasks humans routinely do").
execute a plurality of behavior policy improvement iterations (Abstract idea of analyzing data. Mental process. A human- mind with pen and paper can generate/determine/model data);
providing an initial policy and a lower bound of behavior policy performance, with each subsequent behavior policy improvement iteration (Abstract idea of analyzing data. Mental process. A human can select data/policies);
defining a behaviorally diverse set of behavior policies comprising a plurality of behavior policies which are each safe for control of the artificial agent in the environment and which exceed the lower bound of behavior policy performance, selected using a diverse exploration strategy from a stochastic behavior policy space (encompasses mental observations or evaluations, e.g., a computer programmer’s mental identification of data);
assessing performance of the behaviorally diverse set, by receipt of the data representing the performance of the artificial agent in the environment operating according to each of the plurality of behavior policies of the behaviorally diverse set of behavior policies (encompasses mental observations or evaluations, e.g., a computer programmer’s mental identification of data);
updating the lower bound of behavior policy performance (mental process of modeling and/or performing mathematical operations with assistance of pen and paper).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A system for learning behavior policies for an artificial agent, input port, processor, memory (claim 22) (computer component is recited at a high level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component; the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention." Alice, 134 S. Ct. at 2358);
receive data representing performance of the artificial agent in an environment operating according to a behavior policy (mere data gathering and output recited at a high level of generality - insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A system for learning behavior policies for an artificial agent, input port, processor, memory (claim 22) (computer component is recited at a high level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component; the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention." Alice, 134 S. Ct. at 2358);
receive data representing performance of the artificial agent in an environment operating according to a behavior policy (mere data gathering and output recited at a high level of generality - insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)).
Further, the receiving/transmitting steps were considered to be extra-solution activity in Step 2A Prong 2, and thus it is re-evaluated in Step 2B to determine if it is more than what is well-understood, routine, conventional activity in the field. The receiving and/or transmitting limitations constitute extra-solution activity. See buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355 (Fed. Cir. 2014) ("That a computer receives and sends the information over a network-with no further specification-is not even arguably inventive."). The court decisions cited in MPEP 2106.05(d)(II) indicate that merely Receiving and/or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information). Thereby, a conclusion that the claimed receiving/transmitting steps are well-understood, routine, conventional activity is supported under Berkheimer. The claim is not patent eligible.
23. (New) The system according to claim 22, wherein the at least one automated processor is further configured to select at least one behavior policy for inclusion within the diverse set based on at least a predicted state of the artificial agent operating in the environment according to the selected at least one behavior policy(further expand mental process, it is well-settled that collecting and analyzing information by steps people go through in their minds or by mathematical algorithms, without more, are mental processes in the abstract-idea category. Elec. Power Grp., LLC v. Alstom S.A., 830 F.3d 1350, 1353-54 (Fed. Cir. 2016).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9, 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Eastep (US 2016/0179162) in view of Ghavamzadeh (US 2016/0283970).
1, 21-22. (Currently Amended) A system for learning and deploying behavior policies for an artificial agent (“a hierarchy of identical Reinforcement Learning agents”, 0026, 0027, 0029), comprising:
at least one automated processor (“if the objective function is to maximize performance, then performance of the node may be defined as the minimum performance of any active cores in processors in the node, performance of the cabinet may be defined as the minimum performance of any active node in the cabinet, and performance of the job may be defined as the minimum performance of any active cabinet in the job”, 0037; 0009) configured to iteratively improve a behavior policy of the artificial agent through a plurality of behavior policy improvement iterations, the improved behavior policy in each behavior policy improvement iteration being selected from a plurality of behavior policies in a space of stochastic behavior policies (“the Reinforcement Learning algorithm employs a technique called Stochastic Policy Gradients to navigate efficiently while still achieving good or acceptable results.”, 0027; “stochastic policy may be adopted to avoid “local maxima” pitfalls of simpler gradient methods and to balance tradeoffs of exploration vs. exploitation”, 0030), wherein in each of the plurality of behavior policy improvement iterations:
a diverse exploration strategy (“stochastic policy may be adopted to avoid “local maxima” pitfalls of simpler gradient methods and to balance tradeoffs of exploration vs. exploitation”, 0030) is employed which strives for behavior diversity in the space of stochastic policies, by deploying a diverse set comprising a plurality of behavior policies which are ensured as being safe while excluding inferior behavior policies which do not exceed a lower bound of behavior policy performance determined based on prior assessment of behavior policies during each iteration of policy improvement Eastep (US 2016/0179162) teaches reinforcement learning policy optimizing (“Reinforcement Learning agents optimize policy not through models but by interaction and empirical experimentation with the environment; they continually assess the consequences of actions to adapt their behavior for best results, where the quality of results is defined by an objective function. Experiments are selected in a systematic way to navigate the space of all possible policy options in an efficient manner. According to an embodiment, the Reinforcement Learning algorithm employs a technique called Stochastic Policy Gradients to navigate efficiently while still achieving good or acceptable results”, 0027);
behavior policy performance of each of the plurality of behavior policies policy of the diverse set is assessed with respect to operation of the artificial agent for each iteration of the iterative improvement(“Reinforcement Learning agents optimize policy not through models but by interaction and empirical experimentation with the environment; they continually assess the consequences of actions to adapt their behavior for best results, where the quality of results is defined by an objective function. Experiments are selected in a systematic way to navigate the space of all possible policy options in an efficient manner. According to an embodiment, the Reinforcement Learning algorithm employs a technique called Stochastic Policy Gradients to navigate efficiently while still achieving good or acceptable results”, 0027); and
the lower bound of behavior policy performance is updated based on the assessed behavior policy performance.
Eastep fails to particularly call for the lower bound of behavior policy performance is updated based on the assessed behavior policy performance.
Ghavamzadeh teaches the lower bound of behavior policy performance is updated based on the assessed behavior policy performance (reads on adjusting confidence levels. “adjusts the default confidence level α or otherwise provides the confidence level α”, 0190, 0199).
It would have been obvious to combine the references before the effective filing date because they are in the same field of endeavor and varying limits, bounds, confidence levels allows for optimizing under dynamic conditions and environments.
2. (Currently Amended) The system according to claim 1, wherein each behavior policy has a variance associated with an estimate of its behavior policy performance by importance sampling, and each diverse set has a common average variance, in each of a plurality of behavior policy improvement iterations (“the objective function evaluated by each Reinforcement Learning agent in the H-POMDP will be an aggregation of each child's objective function value. Many aggregations are possible including the minimum of the child objective function values, the average, the variance, etc. In one embodiment, if the objective function is to maximize performance,”, 0037).
3. (Currently Amended) The system according to claim 1, wherein each of the behavior policy performance and behavior policy behavior diversity is quantified according to a common objective function, and the at least one automated processor is further configured to employ the common objective function to assess the behavior policy performance of the diverse set(“the objective function evaluated by each Reinforcement Learning agent in the H-POMDP will be an aggregation of each child's objective function value. Many aggregations are possible including the minimum of the child objective function values, the average, the variance, etc. In one embodiment, if the objective function is to maximize performance,”, 0037).
4. (Currently Amended) The system according to claim 1, wherein the diverse set (reads on e.g., using iterations, and claim does not specify using a plurality of diverse policies at the same time) comprises a plurality of behavior policies predefined upon commencement of a respective single iteration of behavior policy improvement iteration of the plurality of behavior policy improvement iterations, the system further comprising at least one memory configured to store the updated lower bound of behavior policy performance (“There are many ways to determine when a good policy is reached in accordance with some embodiments. One canonical way is convergence testing: a good policy has been reached if the change in policy over the last k iterations has been less than epsilon. k and epsilon are free parameters that may be tuned according to offline manual procedures.”, 0049).
5. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to adaptively define the diverse set based on assessed behavior policy performance within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations(“There are many ways to determine when a good policy is reached in accordance with some embodiments. One canonical way is convergence testing: a good policy has been reached if the change in policy over the last k iterations has been less than epsilon. k and epsilon are free parameters that may be tuned according to offline manual procedures.”, 0049).
6. (Currently Amended) The system according to claim 5, wherein the at least one automated processor is further configured to control the adaptation based on at least a change in the lower bound of behavior policy performance as a selection criterion for a subsequent diverse set within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations(“There are many ways to determine when a good policy is reached in accordance with some embodiments. One canonical way is convergence testing: a good policy has been reached if the change in policy over the last k iterations has been less than epsilon. k and epsilon are free parameters that may be tuned according to offline manual procedures.”, 0049). Ghavamzadeh teaches the lower bound of behavior policy performance is updated based on the assessed behavior policy performance (reads on adjusting confidence levels. “adjusts the default confidence level α or otherwise provides the confidence level α”, 0190, 0199).
7. (Currently Amended) The method system according to claim 5, wherein the at least one automated processor is further configured to control the adaptation based on at least feedback of a system state of the artificial agent received after deploying a prior behavior policy within a respective single behavior policy improvement iteration of the plurality of behavior policy improvement iterations (“while managing the job to a job power cap or other configurable objective functions such as maximum performance, maximum efficiency (e.g., minimum energy-delay product), maximum performance while managing to a job power cap, maximum efficiency while managing to a job power cap, etc. HGPPM techniques are, at least in part, based on a hierarchical feedback-guided control system implemented with a scalable, fast Hierarchical Partially Observable Markov Decision Process (H-POMDP) Reinforcement Learning (RL) method. Such embodiments can mitigate application load imbalance by steering power between hierarchical system domains, and introducing the key capability to coordinate other broader optimizations across software and hardware abstraction boundaries.”, 013).
9. (Currently Amended) The system according to claim 1, wherein the at least one automated processor is further configured to select the plurality of behavior policies of the diverse set within a respective iteration of the plurality of behavior policy improvement iterations according to an aggregate group statistic(“the objective function evaluated by each Reinforcement Learning agent in the H-POMDP will be an aggregation of each child's objective function value. Many aggregations are possible including the minimum of the child objective function values, the average, the variance, etc. In one embodiment, if the objective function is to maximize performance,”, 0037).
Claims 18-20. (Cancelled)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID R VINCENT whose telephone number is (571)272-3080. The examiner can normally be reached ~Mon-Fri 12-8:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at 5712703428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID R VINCENT/Primary Examiner, Art Unit 2123
Prosecution Insights