CTNF 18/536,317 CTNF 93954 DETAILED ACTION 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. This action is responsive to the original application filed on 12/12/2023. Acknowledgment is made with respect to a claim of priority to Japanese Application JP2023-026924 filed on 2/24/2023. 07-30-03-h AIA Claim Interpretation 07-30-03 AIA The following is a quotation of 35 U.S.C. 112(f): (f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph: An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. 07-30-05 The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked. As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph: (A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function; (B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and (C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function. Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function. Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function. Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. 07-30-06 This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “an acquisition portion”, “a storage portion”, “a selection portion”, “a reward calculation portion”, and “an update portion” in claim 1 and its dependents. Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claims 1-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The claim elements “an acquisition portion”, “a storage portion”, “a selection portion”, “a reward calculation portion”, and “an update portion” in claim 1 invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. More specifically, the specification fails to disclose any structure or algorithm for performing the claimed function of the aforementioned elements. See MPEP § 2181 (II)(B) (“In cases involving a special purpose computer-implemented means-plus-function limitation, the Federal Circuit has consistently required that the structure be more than simply a general purpose computer or microprocessor and that the specification must disclose an algorithm for performing the claimed function. See, e.g., Noah Systems Inc. v. Intuit Inc., 675 F.3d 1302, 1312, 102 USPQ2d 1410, 1417 (Fed. Cir. 2012); Aristocrat, 521 F.3d at 1333, 86 USPQ2d at 1239) . For examination purposes, these aforementioned elements will be interpreted to be generic computer components in the form of hardware or software. Claims 2-6 depend on indefinite claim 1, and are rejected under 35 USC § 112(b) by virtue of this dependency. Appropriate correction is required. Claim 1 further recites the limitations “an acquisition portion that acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed … a reward calculation portion that calculates the internal reward and the external reward by using the subset and the state quantity ” (emphasis added). The term “state quantity” is not defined in the claims and is used in two contexts: a state quantity of the apparatus and a state quantity of a target object. The penultimate limitation of claim 1 refers to “the state quantity”, but it is not clear as to which state quantity this limitation refers to. Please explain. For examination purposes, the limitations will be interpreted to mean “an acquisition portion that acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed … a reward calculation portion that calculates the internal reward and the external reward by using the subset and the state quantity of the apparatus or target object ” (emphasis added). Dependent claims 2-6 depend on indefinite claim 1 , and are also rejected under 35 USC § 112(b) by virtue of this dependency. Independent claims 7 and 8 contain the same indefiniteness issues as claim 1, and are also rejected under 35 USC §112(b) for the same reasons as claim 1. Appropriate correction is required. Claim 2 recites the limitation “select the internal reward used in the external reward having high evaluation” (emphasis added). The term “high” in claim 2 is a relative term which renders the claim indefinite. The term “high” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. What constitutes a reward having a high evaluation versus some other kind of evaluation? Please explain. For examination purposes, the limitation will be interpreted to mean “select the internal reward used in the external reward having [[high]] exceed a threshold evaluation value ” (emphasis added). Dependent claims 5-6 depend on indefinite claim 2 , and are also rejected under 35 USC § 112(b) by virtue of this dependency. Appropriate correction is required. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-8 are rejected under 35 U.S.C 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”). When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If it is determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself. Claim 1 Step 1 : The claim recites a reward calculation device; therefore, it is directed to the statutory category of a machine. Step 2A Prong 1 : The claim recites, inter alia: learns by using an external reward which is a reward for solving a given task and an internal reward which is a reward independent of a task: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of learning using external and internal rewards, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally learn how to perform a task such as preparing a recipe using broadly internal and external rewards. a selection portion that selects, based on a selection criterion, one from the subset of the plurality of state quantities stored by the storage portion: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting state quantities based on a criterion, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally select a state quantity for a task. a reward calculation portion that calculates the internal reward and the external reward by using the subset and the state quantity: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of calculating a reward, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally calculate a reward. an update portion that updates the selection criterion of the subset of the state quantity based on the internal reward and the external reward: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of updating selection criteria external and internal rewards, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally update selection criteria using broadly internal and external rewards. Step 2A Prong 2 : The claim does not recite any additional limitations which integrate the abstract idea into a practical application. Specifically, the additional elements consist of “ an acquisition portion ”, “ a storage portion ”, “ a selection portion ”, “ a reward calculation portion ”, “ an update portion ”, “ acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed ”, and “ stores a subset of a plurality of state quantities used for the internal reward ”. The additional elements of “ an acquisition portion ”, “ a storage portion ”, “ a selection portion ”, “ a reward calculation portion ”, “ an update portion ” amount to generic computer components used as a tool to perform an existing process. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer ( see MPEP § 2106.05(f)). The additional elements “ acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed ” and “ stores a subset of a plurality of state quantities used for the internal reward ” insignificant extra-solution activities required for any uses of the abstract ideas ( see MPEP § 2106.05(g)). Thus, even when viewed individually and as an ordered combination, these additional elements do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea. Step 2B : Finally, the claim taken as a whole does not contain an inventive concept which provides significantly more than the abstract idea. The additional elements of “ an acquisition portion ”, “ a storage portion ”, “ a selection portion ”, “ a reward calculation portion ”, “ an update portion ” amount to generic computer components used as a tool to perform an existing process. Thus, the additional elements amount to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer ( see MPEP § 2106.05(f)). The additional elements “ acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed ” and “ stores a subset of a plurality of state quantities used for the internal reward ” insignificant extra-solution activities required for any uses of the abstract ideas ( see MPEP § 2106.05(g)), and are well-understood, routine, conventional activities (see MPEP § 2106.05(d)(II)(i); “Receiving or transmitting data over a network”, “Storing and retrieving information in memory”). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept and thus the claim is subject-matter ineligible. Claim 2 Step 1 : A machine, as above. Step 2A Prong 1 : The claim recites, inter alia: updates the selection criterion so as to select the internal reward used in the external reward having high evaluation: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of updating selection criteria external and internal rewards, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. For example, one can practically and mentally update selection criteria using broadly internal and external rewards. Step 2A Prong 2, Step 2B : The additional element of “ wherein the update portion evaluates the external reward by using a multi-armed bandit model ” amounts to reciting only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is not clear how the multi-armed bandit model is broadly used to evaluate a reward. Thus, the additional element amounts to no more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer ( see MPEP § 2106.05(f)). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible. Claim 3 Step 1 : A machine, as above. Step 2A Prong 1 : The claim recites, inter alia: generates an action of the apparatus by probabilistically selecting one of a policy of the external reward and a policy of the internal reward associated with the subset of the state quantity: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting a policy of the external and internal rewards, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. acquires the state quantity when the apparatus performs the generated action, and based on an acquired result, the selection criterion, and the external reward, selects a policy of an internal reward associated with the subset of the state quantity used for calculation of the internal reward : Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting a policy and acquiring a state quantity, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. Step 2A Prong 2, Step 2B : The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible. Claim 4 Step 1 : A machine, as above. Step 2A Prong 1 : The claim recites the abstract ideas of the preceding claims from which it depends. Step 2A Prong 2, Step 2B : The additional element of “ wherein the selection criterion is any of a UCB value of a UCB (Upper Confidence Bound) method, a probability in a Thompson sampling method, and a likelihood in a MED (minimum Empirical Divergence) method ” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use ( see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible. Claim 5 Step 1 : A machine, as above. Step 2A Prong 1 : The claim recites, inter alia: wherein the update portion generates an action of the apparatus by probabilistically selecting one of a policy of the external reward and a policy of the internal reward associated with the subset of the state quantity, acquires the state quantity when the apparatus performs the generated action, and based on an acquired result, the selection criterion, and the external reward, selects a policy of an internal reward associated with the subset of the state quantity used for calculation of the internal reward: Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of selecting a policy of the external and internal rewards, which is an evaluation or observation that is practically capable of being performed in the human mind with the assistance of pen and paper. Step 2A Prong 2, Step 2B : The claim does not recite any additional elements that are sufficient to integrate the judicial exceptions into a practical application or amount to significantly more than the judicial exception. As such, the claim is ineligible. Claim 6 Step 1 : A machine, as above. Step 2A Prong 1 : The claim recites the abstract ideas of the preceding claims from which it depends. Step 2A Prong 2, Step 2B : The additional element of “ wherein the selection criterion is any of a UCB value of a UCB (Upper Confidence Bound) method, a probability in a Thompson sampling method, and a likelihood in a MED (minimum Empirical Divergence) method ” amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use ( see MPEP § 2106.05(h). Taken alone or in combination, the additional elements of the claim do not provide an inventive concept, integrate the abstract ideas into a practical application, or provide significantly more than the abstract ideas of the claim and thus the claim is subject-matter ineligible. Claim 7 Claim 7 recites a method (step 1: a process) to perform the steps of claim 1, without any additional elements that integrate the abstract ideas into a practical application or provide significantly more than the abstract idea by itself, and is thus rejected for the same reasons set forth in the rejection of claim 1. Claim 8 Claim 8 recites a computer-readable non-transitory recording medium (step 1: a manufacture) using a program and computer to perform the steps of claim 1, which by MPEP 2106.05(f) (“apply it”) cannot integrate an abstract idea into a practical application or provide significantly more than the abstract idea by itself, and is thus rejected for the same reasons set forth in the rejection off claim 1. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1-8 are rejected under 35 U.S.C. § 103 as being obvious over Badia et al. (Badia et al., “Agent57: Outperforming the Atari Human Benchmark”, Mar. 30, 2020, arXiv:2003.13350v1, pp. 1-30, hereinafter “Badia”) in view of Singh et al. (Singh et al., “Intrinsically Motivated Reinforcement Learning”, 2004, Advances in neural information processing systems, vol. 17 (NIPS), MIT Press, Cambridge, MA (2005), pp. 1-8, hereinafter “Singh”). Regarding claim 1 , Badia discloses [a] reward calculation device in a learning apparatus that learns by using an external reward which is a reward for solving a given task and an internal reward which is a reward independent of a task, the reward calculation device comprising: (§1; “NGU achieves this by augmenting the reward signal with an internally generated intrinsic reward that is sensitive to novelty at two levels: short-term novelty within an episode and long-term novelty across episodes … A new parameterization of the state-action value function that decomposes the contributions of the intrinsic and extrinsic rewards. As a result, we significantly increase the training stability over a large range of intrinsic reward scales”, which discloses learning using an external or extrinsic and internal or intrinsic reward ; and §4; the experiments section is inherently implemented using a device in a learning apparatus ; and §3.1; “The sets of weights _e and _i separately parameterize two neural networks with identical architecture and _ = _i [ _e. Both Q(x; a; j; _e) and Q(x; a; j; _i) are optimized separately in the learner with rewards re and ri respectively, but with the same target policy _(x) = arg maxa2A Q(x; a; j; _)”, where r e and r i are the external and internal rewards ) a storage portion that stores a subset of a plurality of state quantities used for the internal reward; (§2; “The per-episode novelty, r episodict , rapidly vanishes over the course of an episode, and it is computed by comparing observations to the contents of an episodic memory”, which discloses an episodic memory M that stores a subset of past embedded state observations ) a selection portion that selects, based on a selection criterion, one from the subset of the plurality of state quantities stored by the storage portion; (§1; “A meta-controller: an adaptive mechanism to select Agent57: Outperforming the Atari Human Benchmark which of the policies (parameterized by exploration rate and discount factors) to prioritize throughout the training process”, which discloses a meta-controller that selects, from among the N-indexed policies, which policy to deploy. Each policy is associated with a distinct weighting of the intrinsic reward computed from the stored state subset ; and §3.2; “Thus, a classical bandit algorithm such as Upper Confidence Bound (UCB; Garivier & Moulines, 2008) will not be able to adapt to the changes of the reward through time” ) a reward calculation portion that calculates the internal reward and the external reward by using the subset and the state quantity; and (§3.1; “The proposed architectural improvement consists in splitting the state-action value function in the following way: Q(x; a; j; _) = Q(x; a; j; _e) + _jQ(x; a; j; _i); … Both Q(x; a; j; _e) and Q(x; a; j; _i) are optimized separately in the learner with rewards re and ri respectively, but with the same target policy _(x) = arg maxa2A Q(x; a; j; _)”, wherein the “subset” is the stored episodic memory of state embeddings, and “the state quantity” is the currently acquired live observation x. Further, the external reward r e is calculated using the current observation x and the internal reward r i is calculated using the episodic memory comparison ; and §2; “The per-episode novelty, r episodict , rapidly vanishes over the course of an episode, and it is computed by comparing observations to the contents of an episodic memory” ) an update portion that updates the selection criterion of the subset of the state quantity based on the internal reward and the external reward (§3.2; “The undiscounted extrinsic episode returns, noted Rek(Jk), are used as a reward signal to train the multi-arm bandit algorithm of the meta-controller”, which discloses updating a selection criterion or UCB selection criterion based on the internal and external rewards, and the “undiscounted returns” are the total episode returns under the selected policy. The bandit update is driven by a signal that jointly incorporates both the internal and external rewards and §2; “r j,t = r e t +β j r i t ”. ). Badia fails to explicitly disclose but Singh discloses an acquisition portion that acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed; (§4; “The agent has an eye, a hand, and a visual marker (seen as a cross hair in the figure). The agent’s sensors tell it what objects (if any) are under the eye, hand and marker.”, which discloses that an RL agent that acquire both the state of the apparatus (the agent’s eye, hand, and marker positions) and the state of the distinct target objects (light witch, ball, blocks, etc.), and these states are acquired in the experiments of Singh ; and Figure 2, Learning Algorithm; the figure and algorithm disclose the obtaining or acquiring of state information ). Badia and Singh are analogous art because both are concerned with reinforcement learning using multiple rewards. Before the effective filing date of the claimed invention, it would have been obvious to one skilled in reinforcement learning to combine the acquiring of apparatus and target object state quantities of Singh with the device of Badia to yield to the predictable result of an acquisition portion that acquires a state quantity of the apparatus which performs the task and a state quantity of a target object to which the task is performed . The motivation for doing so would be to allow artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy (Singh; Abstract). Regarding claim 7 , it is a method claim corresponding to the steps of claim 1, and is rejected for the same reasons as claim 1. Regarding claim 8 , it is a computer-readable non-transitory recording medium claim corresponding to the steps of claim 1, and is rejected for the same reasons as claim 1. Regarding claim 2 , the rejection of claim 1 is incorporated and Badia discloses wherein the update portion evaluates the external reward by using a multi-armed bandit model and updates the selection criterion so as to select the internal reward used in the external reward having high evaluation (§3.2; “We propose to implement the meta-controller using a non stationary multi-arm bandit algorithm running independently on each actor … Each arm j from the N-arm bandit is linked to a policy in the family and corresponds to a pair (βj,γj). … The undiscounted extrinsic episode returns, noted Re k(Jk), are used as a reward signal to train the multi-arm bandit algorithm of the meta-controller”. Because arm j encodes the intrinsic reward weighting, selecting the arm with the highest UCB score is selecting the internal reward policy associated with the high external reward evaluation ). Regarding claim 3 , the rejection of claim 1 is incorporated and Badia discloses wherein the update portion generates an action of the apparatus by probabilistically selecting one of a policy of the external reward and a policy of the internal reward associated with the subset of the state quantity, acquires the state quantity when the apparatus performs the generated action, and based on an acquired result, the selection criterion, and the external reward, selects a policy of an internal reward associated with the subset of the state quantity used for calculation of the internal reward (§3.2; “At the beginning of each episode, say, the k-th episode, the meta-controller chooses an arm Jk setting which policy will be executed. We use capital letters for the arm Jk because it is a random variable. Then the l-th actor acts l-greedily with respect to the corresponding state-action value function, Q(x,a,Jk;θl), for the whole episode. The undiscounted extrinsic episode returns, noted Re k(Jk), are used as a reward signal to train the multi-arm bandit algorithm of the meta-controller. The reward signal Re k(Jk) is non-stationary, as the agent changes throughout training. Thus, a classical bandit algorithm such as Upper Confidence Bound (UCB; Garivier & Moulines, 2008) will not be able to adapt to the changes of the reward through time. Therefore, we employ a simplified sliding-window UCB with UCB-greedy exploration. With probability 1−UCB, this algorithm runs a slight modification of classic UCB on a sliding window of size τ and selects a random arm with probability UCB” ). Regarding claim 4 , the rejection of claim 1 is incorporated and Badia discloses wherein the selection criterion is any of a UCB value of a UCB (Upper Confidence Bound) method, a probability in a Thompson sampling method, and a likelihood in a MED (minimum Empirical Divergence) method (§3.2; “Therefore, we employ a sim plified sliding-window UCB with UCB-greedy exploration. With probability 1−UCB, this algorithm runs a slight modification of classic UCB on a sliding window of size τ and selects a random arm with probability UCB” ). Regarding claim 5 , the rejection of claims 1 and 2 are incorporated and Badia discloses wherein the selection criterion is any of a UCB value of a UCB (Upper Confidence Bound) method, a probability in a Thompson sampling method, and a likelihood in a MED (minimum Empirical Divergence) method (§3.2; “At the beginning of each episode, say, the k-th episode, the meta-controller chooses an arm Jk setting which policy will be executed. We use capital letters for the arm Jk because it is a random variable. Then the l-th actor acts l-greedily with respect to the corresponding state-action value function, Q(x,a,Jk;θl), for the whole episode. The undiscounted extrinsic episode returns, noted Re k(Jk), are used as a reward signal to train the multi-arm bandit algorithm of the meta-controller. The reward signal Re k(Jk) is non-stationary, as the agent changes throughout training. Thus, a classical bandit algorithm such as Upper Confidence Bound (UCB; Garivier & Moulines, 2008) will not be able to adapt to the changes of the reward through time. Therefore, we employ a simplified sliding-window UCB with UCB-greedy exploration. With probability 1−UCB, this algorithm runs a slight modification of classic UCB on a sliding window of size τ and selects a random arm with probability UCB” ). Regarding claim 6 , the rejection of claims 1 and 2 are incorporated and Badia discloses wherein the selection criterion is any of a UCB value of a UCB (Upper Confidence Bound) method, a probability in a Thompson sampling method, and a likelihood in a MED (minimum Empirical Divergence) method (§3.2; “The undiscounted extrinsic episode returns, noted Re k(Jk), are used as a reward signal to train the multi-arm bandit algorithm of the meta-controller. … Therefore, we employ a simplified sliding-window UCB with UCB-greedy exploration. With probability 1−UCB, this algorithm runs a slight modification of classic UCB on a sliding window of size τ and selects a random arm with probability UCB” ). Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to Brent Hoover whose telephone number is (303)297-4403. The examiner can normally be reached Monday - Friday 9-5 MST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Kawsar can be reached at 571-270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /BRENT JOHNSTON HOOVER/Primary Examiner, Art Unit 2127 Application/Control Number: 18/536,317 Page 2 Art Unit: 2127 Application/Control Number: 18/536,317 Page 3 Art Unit: 2127 Application/Control Number: 18/536,317 Page 4 Art Unit: 2127 Application/Control Number: 18/536,317 Page 5 Art Unit: 2127 Application/Control Number: 18/536,317 Page 6 Art Unit: 2127 Application/Control Number: 18/536,317 Page 7 Art Unit: 2127 Application/Control Number: 18/536,317 Page 8 Art Unit: 2127 Application/Control Number: 18/536,317 Page 9 Art Unit: 2127 Application/Control Number: 18/536,317 Page 10 Art Unit: 2127 Application/Control Number: 18/536,317 Page 11 Art Unit: 2127 Application/Control Number: 18/536,317 Page 12 Art Unit: 2127 Application/Control Number: 18/536,317 Page 13 Art Unit: 2127 Application/Control Number: 18/536,317 Page 14 Art Unit: 2127 Application/Control Number: 18/536,317 Page 15 Art Unit: 2127 Application/Control Number: 18/536,317 Page 16 Art Unit: 2127 Application/Control Number: 18/536,317 Page 17 Art Unit: 2127 Application/Control Number: 18/536,317 Page 18 Art Unit: 2127 Application/Control Number: 18/536,317 Page 19 Art Unit: 2127 Application/Control Number: 18/536,317 Page 20 Art Unit: 2127 Application/Control Number: 18/536,317 Page 21 Art Unit: 2127 Application/Control Number: 18/536,317 Page 22 Art Unit: 2127 Application/Control Number: 18/536,317 Page 23 Art Unit: 2127 Application/Control Number: 18/536,317 Page 24 Art Unit: 2127 Application/Control Number: 18/536,317 Page 25 Art Unit: 2127
