DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
2. Receipt of Applicant’s preliminary amendment filed on 02/01/2024 is acknowledged. The preliminary amendment includes the amending of the specification and the amending of claims 7-15 and 18-20.
Information Disclosure Statement
3. The information disclosure statements (IDS) submitted on 02/01/2024 and 04/23/2025 have been received, entered into the record, and considered. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claims (1-15) and (16-20) are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Under the 2019 PEG, when considering subject matter eligibility under 35 U.S.C. § 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (step 1). If the claim does fall within one of the statutory categories, it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea) (step 2A prong 1), and if so, it must additionally be determined whether the claim is integrated into a practical application (step 2A prong 2). If an abstract idea is present in the claim without integration into a practical application, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself (step 2B).
In the instant case, claims (1-15) and (16-20) are directed to a method and system. Thus, each of the claims falls within one of the four statutory categories. However, the claims also fall within the judicial exception of an abstract idea.
Under Step 2A Prong 1, the test is to identify whether the claims are “directed to” a judicial exception. The examiner notes that the claimed invention is directed to an abstract idea in that the instant application is directed to mental processes, specifically selecting an action.
The examiner further notes that claims (1-15) and (16-20) recite a method and system for selecting an action which is similar to themes defined above of method of mental processes such as selecting an action, and is similar to the abstract idea identified in the 2019 PEG in grouping “c” in that the claims recite certain methods of mental processes such as selecting an action. The limitations, substantially comprising the body of the claim, recite a process of selecting an action. The examiner notes that the claimed invention selects an action. Because the limitations above closely follow the steps in selecting an action, and the steps of the claims involve mental processes, the claim recites an abstract idea consistent with the “mental processes” grouping set forth in the 2019 PEG.
Claim 1:
A method of selecting at least one action from a plurality of actions to be performed in an environment, the method comprising: maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed and a difference between the number of times the action has been performed and a number of observed resulting rewards for the action, each reward being a numeric value that measures an outcome of a given action;
determining, from the count data and a bandit score provided by a bandit model, an expected score for each action from the plurality of actions;
the bandit score provided by the bandit model for a given history of performed actions and observed rewards; and
the expected score determined by determining an expected value of the bandit score given a likelihood of some of the plurality of actions having unobserved pending rewards; and
selecting, from the plurality of actions and based on the expected score for each action, the at least one action to be performed in the environment.
These limitations, as drafted, is an apparatus that, under its broadest reasonable interpretation, covers the performance of mental processes specifically selecting an action. Selecting an action has long before the modern computer was invented, and continues to be predominantly a product of human endeavor. The instant application is directed to selecting an action. Moreover, the determination of a defined expected score can be performed by a human via their mind and/or pen & paper. Additionally, the selection of an action can be performed by a human via their mind and/or pen & paper. Because the limitations above closely follow the steps of selecting an action, and the steps involved human judgments, observations and evaluations that can be practically or reasonably performed in the human mind and/or pen & paper, the claim recites an abstract idea consistent with the “mental process” grouping set forth in the 2019 PEG.
If the claims are directed toward the judicial exception of an abstract idea, it must then be determined under Step 2A Prong 2 whether the judicial exception is integrated into a practical application. The Examiner notes that considerations under Step 2A Prong 2 comprise most the consideration previously evaluated in the context of Step 2B. The Examiner submits that the considerations discussed previously determined that the claim does not recite “significantly more” at Step 2B would be evaluated the same under Step 2A Prong 1 and result in the determination that the claim does not integrate the abstract idea into a practical application.
The instant application fails to integrate the judicial exception into a practical application because the instant application merely recites words “apply it” (or an equivalent) with the judicial exception or merely includes instructions to implement an abstract idea. The instant application is directed to an apparatus instructing the reader to implement the identified apparatus of mental processes of selecting an action. The elements of the claim do not themselves amount to an improvement to the computer, to a technology or another technical field. Moreover, the maintaining of different types of data is a data storage operation that is an insignificant data storage operation that does not integrate the abstract idea into a practical application. Furthermore, the output of a defined bandit score is a data output operation that is an insignificant data output operation that does not integrate the abstract idea into a practical application.
Here, the claim elements entirely comprise the abstract idea, leaving little if any aspects of the claim for further consideration under Step 2A Prong 2. In short, the claims have failed to integrate a practical application (see at least 84 Fed. Reg. (4) at 55). Under the 2019 PEG, this supports the conclusion that the claim is directed to an abstract idea, and the analysis proceeds to Step 2B.
While many considerations in Step 2A need not be reevaluated in Step 2B because the outcome will be the same. Here, on the basis of the additional elements other than the abstract idea, considered individually and in combination as discussed above, the Examiner respectfully submits that the claim 1 does not contain any additional elements that individually or as an ordered combination amount to an inventive concept and the claims are ineligible.
With respect to the dependent claims do not recite anything that is found to render the abstract idea as being transformed into a patent eligible invention. The dependent claims are merely reciting further embellishments of the abstract idea and do not claim anything that amounts to significantly more than the abstract idea itself.
With respect to the dependent claims, they have been considered and are not found to be reciting anything that amounts to being significantly more than the abstract idea. Claims 2-15 are directed to further embellishments of the central theme of the abstract idea in that the claims are directed to further embellishments of the selecting an action of the steps of claim 1 and do not amount to significantly more.
Specifically, claim 2 recites the selection of an action which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Furthermore, claim 3 recites the selection of at least one of defined multiple actions which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Additionally, claim 4 recites the selection of at least one of defined multiple actions which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Moreover, claim 5 recites the selection of an action which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Furthermore, claim 6 recites the selection of at least one of defined multiple actions which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Additionally, claim 7 recites the updating of count data which can be performed by the human mind and/or pen & paper and does not amount to significantly more. Moreover, the receiving of an indication is a data gathering operation that is an insignificant data gathering operation that does not integrate the abstract idea into a practical application.
Moreover, claim 8 recites the maintaining of defined count data which is still a data storage operation that is an insignificant data storage operation that does not integrate the abstract idea into a practical application.
Furthermore, claim 9 recites the use of a Whittle index to determine a bandit score which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Additionally, claim 10 recites the use of an infinite time horizon algorithm to determine a bandit score which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Moreover, claim 11 recites the determination of a defined expected score via determined configurations, probabilities, and multiplication operations which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Furthermore, claim 12 recites the use of one of probabilistic sampling and Monte Carlo simulation which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Additionally, claim 13 recites the defining of a rewards measure which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Moreover, claim 14 recites the defining of a rewards measure which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Furthermore, claim 15 recites the averaging of an expected score and subsequent selection of a highest average score which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Claim 16:
A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for selecting at least one action from a plurality of actions to be performed in an environment, the operations comprising: maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed and a difference between the number of times the action has been performed and a number of observed resulting rewards for the action, each reward being a numeric value that measures an outcome of a given action;
determining, from the count data and a bandit score provided by a bandit model, an expected score for each action from the plurality of actions;
the bandit score provided by the bandit model for a given history of performed actions and observed rewards; and
the expected score determined by determining an expected value of the bandit score given a likelihood of some of the plurality of actions having unobserved pending rewards; and
selecting, from the plurality of actions and based on the expected score for each action, the at least one action to be performed in the environment.
These limitations, as drafted, is an apparatus that, under its broadest reasonable interpretation, covers the performance of mental processes specifically selecting an action. Selecting an action has long before the modern computer was invented, and continues to be predominantly a product of human endeavor. The instant application is directed to selecting an action. Moreover, the determination of a defined expected score can be performed by a human via their mind and/or pen & paper. Additionally, the selection of an action can be performed by a human via their mind and/or pen & paper. Because the limitations above closely follow the steps of selecting an action, and the steps involved human judgments, observations and evaluations that can be practically or reasonably performed in the human mind and/or pen & paper, the claim recites an abstract idea consistent with the “mental process” grouping set forth in the 2019 PEG.
The mere nominal recitation of generic computing components such as “one or more computers” and “one or more storage devices” do not take the claim out of certain methods of mental processes grouping. Therefore, the limitation is directed to an abstract idea.
If the claims are directed toward the judicial exception of an abstract idea, it must then be determined under Step 2A Prong 2 whether the judicial exception is integrated into a practical application. The Examiner notes that considerations under Step 2A Prong 2 comprise most the consideration previously evaluated in the context of Step 2B. The Examiner submits that the considerations discussed previously determined that the claim does not recite “significantly more” at Step 2B would be evaluated the same under Step 2A Prong 1 and result in the determination that the claim does not integrate the abstract idea into a practical application.
The instant application fails to integrate the judicial exception into a practical application because the instant application merely recites words “apply it” (or an equivalent) with the judicial exception or merely includes instructions to implement an abstract idea. The instant application is directed to an apparatus instructing the reader to implement the identified apparatus of mental processes of selecting an action. The elements of the claim do not themselves amount to an improvement to the computer, to a technology or another technical field. Moreover, the maintaining of different types of data is a data storage operation that is an insignificant data storage operation that does not integrate the abstract idea into a practical application. Furthermore, the output of a defined bandit score is a data output operation that is an insignificant data output operation that does not integrate the abstract idea into a practical application.
Here, the claim elements entirely comprise the abstract idea, leaving little if any aspects of the claim for further consideration under Step 2A Prong 2. In short, the claims have failed to integrate a practical application (see at least 84 Fed. Reg. (4) at 55). Under the 2019 PEG, this supports the conclusion that the claim is directed to an abstract idea, and the analysis proceeds to Step 2B.
While many considerations in Step 2A need not be reevaluated in Step 2B because the outcome will be the same. Here, on the basis of the additional elements other than the abstract idea, considered individually and in combination as discussed above, the Examiner respectfully submits that the claim 16 does not contain any additional elements that individually or as an ordered combination amount to an inventive concept and the claims are ineligible.
With respect to the dependent claims do not recite anything that is found to render the abstract idea as being transformed into a patent eligible invention. The dependent claims are merely reciting further embellishments of the abstract idea and do not claim anything that amounts to significantly more than the abstract idea itself.
With respect to the dependent claims, they have been considered and are not found to be reciting anything that amounts to being significantly more than the abstract idea. Claims 17-20 are directed to further embellishments of the central theme of the abstract idea in that the claims are directed to further embellishments of the selecting an action of the steps of claim 16 and do not amount to significantly more.
Specifically, claim 17 recites the updating of count data which can be performed by the human mind and/or pen & paper and does not amount to significantly more. Moreover, the receiving of an indication is a data gathering operation that is an insignificant data gathering operation that does not integrate the abstract idea into a practical application.
Furthermore, claim 18 recites the maintaining of defined count data which is still a data storage operation that is an insignificant data storage operation that does not integrate the abstract idea into a practical application
Additionally, claim 19 recites the determination of a defined expected score via determined configurations, probabilities, and multiplication operations which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Moreover, claim 20 recites the use of one of probabilistic sampling and Monte Carlo simulation which can be performed by the human mind and/or pen & paper and does not amount to significantly more.
Claim Rejections - 35 USC § 103
6. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
9. Claims 1, 7-8, 12-13, 15-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey et al. (Article entitled “Multi-armed Bandit Problems with Dependent Arms”, dated 2007), in view of Jiang et al. (U.S. PGPUB 2022/0070698), and further in view of Guha et al. (Article entitled “Multi-Armed Bandit Problems with Delayed Feedback”, dated 18 June 2013).
10. Regarding claims 1 and 16, Pandey teaches a method and system comprising:
A) maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed (Page 725, Section 5.2); and
C) each reward being a numeric value that measures an outcome of a given action (Page 722, Section 3);
D) determining, from the count data and a bandit score provided by a bandit model, an expected score for each action from the plurality of actions (Page 722, Section 3);
E) the bandit score provided by the bandit model for a given history of performed actions and observed rewards (Page 723, Section 3, Page 726, Section 5.3); and
G) selecting, from the plurality of actions and based on the expected score for each action, the at least one action to be performed in the environment (Page 723, Section 3, Page 726, Section 5.3, Page 728, Section 7).
The examiner notes that Pandey teaches “maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed” as “In this paper, we set the policy…Ti is the number of arm pulls for i” (Page 725, Section 5.2). The examiner further notes that the maintaining of the number of arm pulls teaches the claimed number of times an action has been performed in the broadest reasonable interpretation. The examiner further notes that Pandey teaches “each reward being a numeric value that measures an outcome of a given action” as “In each timestep t, one arm i must be chosen (“pulled”), and it emits a reward R(t) which is 1 with probability θi, and 0 otherwise. The objective is to pull arms so as to maximize the expected discounted reward” (Page 722, Section 3). The examiner further notes a reward function that outputs a numeric value (See examples of 1 or 0) teachers the claimed reward in the broadest reasonable interpretation. The examiner further notes that Pandey teaches “determining, from the count data and a bandit score provided by a bandit model, an expected score for each action from the plurality of actions” as “In each timestep t, one arm i must be chosen (“pulled”), and it emits a reward R(t) which is 1 with probability θi, and 0 otherwise. The objective is to pull arms so as to maximize the expected discounted reward… where 0 < α… for a given time horizon T. Maximizing the objective function is equivalent to minimizing the expected regret” (Page 722, Section 3) and “Under the MEAN strategy, the expected reward estimate for each cluster i is given by… grows monotonically towards µi for increasing Ti…marginally in this setting” (Page 726, Section 5.3). The examiner further notes that a calculated estimate teaches the claimed undefined expected score in the broadest reasonable interpretation. Such a calculation is based on the undefined claimed bandit scores score and count data. The examiner further notes that Pandey teaches “the bandit score provided by the bandit model for a given history of performed actions and observed rewards” as “Typically, algorithms for bandit problems iterate over two steps, as shown in Figure 2. For the independent bandit, the update step needs to look only at the pulls and rewards of each arm in isolation. For the dependent bandit, the update step involves computing π[i](t) given data on all prior arm pulls and corresponding rewards from each cluster; but this is typically a well understood statistical procedure” (Page 723, Section 3). The examiner further notes the use of prior arm pulls and corresponding rewards (i.e. the claimed history of performed actions and observed rewards) in calculation of a bandit score teaches the claimed bandit score. The examiner further notes that Pandey teaches “selecting, from the plurality of actions and based on the expected score for each action, the at least one action to be performed in the environment” as “In each timestep t, one arm i must be chosen (“pulled”), and it emits a reward R(t) which is 1 with probability θi, and 0 otherwise. The objective is to pull arms so as to maximize the expected discounted reward… where 0 < α… for a given time horizon T. Maximizing the objective function is equivalent to minimizing the expected regret” (Page 722, Section 3), “Under the MEAN strategy, the expected reward estimate for each cluster i is given by… grows monotonically towards µi for increasing Ti…marginally in this setting” (Page 726, Section 5.3), and “For discounted reward, we generalize the well known Gittins theorem and show that optimal allocation rules are obtained by decoupling the problem in terms of clusters instead of individual arms. For finite horizon reward, we provide a general two-stage allocation policy that selects a cluster followed by an arm in the selected cluster. We provide several instances of our general policy and also provide theoretical justifications on how the regret depends on the characteristics of the clusters. Empirically, we demonstrate the efficacy of our methods and show that they significantly outperform classical ban dit solutions using real-world and synthetic data” (Page 728, Section 7). The examiner further notes that selecting a specific arm (i.e. action) is based off of ascertained expected rewards.
Pandey does not explicitly teach:
B) a difference between the number of times the action has been performed and a number of observed resulting rewards for the action.
Jiang, however, teaches “a difference between the number of times the action has been performed and a number of observed resulting rewards for the action” as “the counting can be performed on either the LBT failures or the LBT successes, and a total number of LBTs initiated can be counted. The total number of LBTs initiated minus the number of LBT successes is the number of LBT failures” (Paragraph 95).
The examiner further notes that although Pandey clearly teaches “count” data of the total number of pulls (i.e. number of times an action has been performed) and the number of successes, there is no explicit teaching of counting the mathematical difference between the two values. However, Jiang teaches the count data of the difference between a total number value and a number of success value. The combination would result in storing such difference data in Pandey.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Jiang’s would have allowed Pandey’s to provide a method for counting various types of data based off of needs, as noted by Jiang (Paragraph 93).
Pandey and Jiang do not explicitly teach:
F) the expected score determined by determining an expected value of the bandit score given a likelihood of some of the plurality of actions having unobserved pending rewards.
Guha, however, teaches “the expected score determined by determining an expected value of the bandit score given a likelihood of some of the plurality of actions having unobserved pending rewards” as “We show that bandit problems with delayed feedback that arise in allocation settings can be forced to have significant structure that gives us the ability to reason about this policy. We show a O(1) approximation for a significantly general class of priors. The structural insights we develop are of key interest and carry over to the setting where the feedback of an action is available instantaneously. In particular, we show a simple 2-approximation for the finite horizon Bayesian bandit problem, improving and generalizing prior work” (Abstract), “We now concretely define the Bayesian multi-armed bandit problem with delayed feedback. There is a bandit with n independent arms. When arm i is played, the reward is drawn i.i.d. from distribution Di, which is unknown. However, a prior Di is specified over possible Di. When arm i is played, the feedback about its reward outcome is learned only after δi steps. The set of observed outcomes so far resolves the prior to a posterior distribution according to Bayes’ rule. We also assume each arm has budget Bi on the maximum reward it can accrue; further plays do not accrue reward. In this setting, the goal is to design a decision policy for allocating the plays to the arms. A decision policy is a mapping from the current state, determined by the posterior distributions of each arm; the plays for which feedback is outstanding; and the remaining budgets of each arm to an action of which arm to play. There is a horizon of T steps, and our goal is to design a polynomial time algorithm that outputs a decision policy maximizing the expected revenue” (Page 1, Section 1.1), “There is a bandit with n independent arms. The arm i underlying reward distribution Di, which is a random variable drawn from a prior distribution Di. These priors are specified as input. The maximum possible reward that can be extracted from this arm is Bi, and though additional plays can be made, they do not accrue additional reward. If an arm is played, the feedback about the reward outcome is available only after δi time steps. As observations are available the successive posteriors (which serve as priors for the next trial), are produced by the Bayes’ Rule. There is a time horizon of T plays. A decision policy specifies which arm to play given the current state of each arm, which is captured by the remaining budget and time horizon, posterior distribution, and plays with outstanding feedback for that arm. Each decision policy has a unique expected reward value that is obtained on executing it (with different execution trajectories differing due to different realizations of the underlying {Di}). Our goal is to design a poly-time algorithm, which outputs a policy that approximately maximizes the expected value derived over the horizon of T plays” (Page 4, Section 2).
The examiner further notes that Guha teaches the concept of an expected score being derived based off of delayed feedback (i.e. unobserved rewards). The combination would in the use of such unobserved rewards in the expected score of Pandey.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Guha’s would have allowed Pandey’s and Jiang’s to provide a method for avoiding allocation problems, as noted by Guha (Page 1, Section 1).
Regarding claims 7 and 17, Pandey further teaches a method and system comprising:
A) receiving an indication that a reward was observed in response to a selected one of the plurality of actions being performed (Page 722, Section 3, Page 725, Section 5.2); and
B) in response, updating the count data (Page 722, Section 3, Page 725, Section 5.2).
The examiner notes that Pandey teaches “maintaining, for each action from the plurality of actions, count data indicative of a number of times the action has been performed” as “In particular, let si(t) be the number of times arm i generated a unit reward when pulled (“successes”), and fi(t) the number of “failures.”” (Page 722, Section 3), “In this paper, we set the policy…Ti is the number of arm pulls for i” (Page 725, Section 5.2). The examiner further notes that the maintaining of the number of arm pulls, number of successes, and number of failures entails receiving an indication of a reward. The examiner further notes that Pandey teaches “in response, updating the count data” as “In particular, let si(t) be the number of times arm i generated a unit reward when pulled (“successes”), and fi(t) the number of “failures.”” (Page 722, Section 3), “In this paper, we set the policy…Ti is the number of arm pulls for i” (Page 725, Section 5.2). The examiner further notes that the maintaining of the number of arm pulls, number of successes, and number of failures entails updating the count data.
Regarding claims 8 and 18, Pandey further teaches a method and system comprising:
B) the count being: a windowed count that counts how many times a reward has been observed for a given action in response to the action being performed during a recent time window that includes a fixed number of most recent time steps (Page 722, Section 3, Page 725, Section 5.2).
The examiner notes that Pandey teaches “the count being: a windowed count that counts how many times a reward has been observed for a given action in response to the action being performed during a recent time window that includes a fixed number of most recent time steps” as “In particular, let si(t) be the number of times arm i generated a unit reward when pulled (“successes”), and fi(t) the number of “failures.”… In each timestep t, one arm i must be chosen (“pulled”), and it emits a reward R(t) which is 1 with probability θi, and 0 otherwise” (Page 722, Section 3), “In this paper, we set the policy…Ti is the number of arm pulls for i” (Page 725, Section 5.2). The examiner further notes that the maintaining of the number of arm pulls, number of successes, and number of failures are for a specific previous “window” of timesteps t.
Pandey does not explicitly teach:
A) wherein maintaining the count data comprises maintaining a count of the difference between the number of times the action has been performed and the number of resulting rewards have been observed for a given action.
Jiang, however, teaches “wherein maintaining the count data comprises maintaining a count of the difference between the number of times the action has been performed and the number of resulting rewards have been observed for a given action” as “the counting can be performed on either the LBT failures or the LBT successes, and a total number of LBTs initiated can be counted. The total number of LBTs initiated minus the number of LBT successes is the number of LBT failures” (Paragraph 95).
The examiner further notes that although Pandey clearly teaches “count” data of the total number of pulls (i.e. number of times an action has been performed) and the number of successes, there is no explicit teaching of counting the mathematical difference between the two values. However, Jiang teaches the count data of the difference between a total number value and a number of success value. The combination would result in storing such difference data in Pandey.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Jiang’s would have allowed Pandey’s to provide a method for counting various types of data based off of needs, as noted by Jiang (Paragraph 93).
Regarding claims 12 and 20, Pandey further teaches a method and system comprising:
A) wherein the bandit score is determined using one of probabilistic sampling and Monte Carlo simulation, further wherein a probabilistic sample is defined as simulating a sequence of actions for missing rewards in the count data, updating a prior expected action reward distribution after each simulated action, and determining the bandit score using simulated rewards when no missing rewards remain, and the probabilistic sampling using a number of probabilistic samples to create an estimate of the expected score (Page 726, Section 5.2).
The examiner notes that Pandey teaches “wherein the bandit score is determined using one of probabilistic sampling and Monte Carlo simulation, further wherein a probabilistic sample is defined as simulating a sequence of actions for missing rewards in the count data, updating a prior expected action reward distribution after each simulated action, and determining the bandit score using simulated rewards when no missing rewards remain, and the probabilistic sampling using a number of probabilistic samples to create an estimate of the expected score” as “Our final strategy, called PMAX, achieves this by computing the posterior distribution of the maximum success probability among all the arms in Ci, given all observations from the cluster. Where analytic formulas for the posterior are not available, Monte Carlo sampling is used” (Page 726, Section 5.2). The examiner further notes that the Monte Carlo sampling teaching the claimed use of Monte Carlo.
Regarding claim 13, Pandey further teaches a method comprising:
A) wherein the rewards measure discrete outcomes (Page 722, Section 3).
The examiner notes that Pandey teaches “wherein the rewards measure discrete outcomes” as “In particular, let si(t) be the number of times arm i generated a unit reward when pulled (“successes”), and fi(t) the number of “failures.”” (Page 722, Section 3). The examiner further notes that successes and failures are discrete values for rewards.
Regarding claim 15, Pandey further teaches a method comprising:
A) wherein the expected score is an average expected score based on a number N of samples (Page 725, Section 5.2); and
B) further wherein selecting the at least one action to be performed in the environment comprises selecting a highest average expected score (Page 725, Section 5.2).
The examiner notes that Pandey teaches “wherein the expected score is an average expected score based on a number N of samples” as “Based on these estimates, TLP chooses a cluster c(t) by running POL on the cluster-arms. Then, it chooses an arm from within cluster c(t) using POL again, using the mean and variance of the success probability θi of each arm i as its reward and variance estimate” (Page 725, Section 5.2) and “Intuitively, to minimize regret, we must quickly find the best arm, and hence the cluster containing that arm. The cluster reward estimate ˆri should tell us the expected maximum success probability of all arms in the cluster, so that the best cluster is chosen in step 2 of TLP as often as possible. A good reward estimate must be accurate and converge quickly (i.e., ˆσi → 0 quickly). We propose three such strategies below. The MEAN strategy is the simplest: it sets ˆri to the average success rate of arms in the cluster” (Page 725, Section 5.2). The examiner further notes that the mean calculation teaches the claimed average expected score. The examiner further notes that Pandey teaches “further wherein selecting the at least one action to be performed in the environment comprises selecting a highest average expected score” as “Based on these estimates, TLP chooses a cluster c(t) by running POL on the cluster-arms. Then, it chooses an arm from within cluster c(t) using POL again, using the mean and variance of the success probability θi of each arm i as its reward and variance estimate” (Page 725, Section 5.2) and “Intuitively, to minimize regret, we must quickly find the best arm, and hence the cluster containing that arm. The cluster reward estimate ˆri should tell us the expected maximum success probability of all arms in the cluster, so that the best cluster is chosen in step 2 of TLP as often as possible. A good reward estimate must be accurate and converge quickly (i.e., ˆσi → 0 quickly). We propose three such strategies below. The MEAN strategy is the simplest: it sets ˆri to the average success rate of arms in the cluster” (Page 725, Section 5.2). The examiner further notes that the selection of the arm (i.e. action) is selected based off of the mean.
11. Claims 2-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey et al. (Article entitled “Multi-armed Bandit Problems with Dependent Arms”, dated 2007), in view of Jiang et al. (U.S. PGPUB 2022/0070698), and further in view of Guha et al. (Article entitled “Multi-Armed Bandit Problems with Delayed Feedback”, dated 18 June 2013) as applied to claims 1, 7-8, 12-13, 15-18, and 20 above, and further in view of Ito (U.S. PGPUB 2024/0103812).
12. Regarding claim 2, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein selecting the at least one action comprises selecting a resource allocation to implement in a telecommunications environment.
Ito, however, teaches “wherein selecting the at least one action comprises selecting a resource allocation to implement in a telecommunications environment” as “The following description will consider the problem of determining a discount coupon to be provided to a customer by an operating company of a certain electronic commerce site. In this case, the action of determining the discount coupon to be provided to a plurality of customers is expressed by a vector a t components of which are the types of the discount coupons to be provided to the customers. For example, an action of providing a discount coupon of a product 1 to a customer A, providing a discount coupon of a product 2 to a customer B, and providing a discount coupon of a product 3 to a customer C is expressed by a vector a.sub.t =(1,2,3, . . . ). Then, it is assumed that a loss l.sub.t.sup.Ta.sub.t is obtained as a feedback. Here, the loss ltTat may be a value based on whether the discount coupon is used, a gaze time, whether the discount coupon has been clicked, a purchase price of a product, a purchase probability, a purchase price, and the like. In this case, application of the above-described information processing method S1 makes it possible to determine a discount coupon that reduces a loss. In particular, even in a case where customer's preferences and utilities tend to change, such as online marketing, it is possible to provide an optimal discount coupon for each customer” (Paragraph 83).
The examiner further notes that Ito teaches an action of determining a coupon (i.e. the undefined resource allocation in the broadest reasonable interpretation) for a webpage on the internet (i.e. the claimed telecommunications environment) in a Bandit setting. The combination would result in the bandit setting of Pandey and Guha to also be in the context of resource allocation in a telecommunications environment.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Ito’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a bandit solution in differing problems, as noted by Ito (Paragraph 82).
Regarding claim 3, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein selecting the at least one action comprises selecting at least one of a drug to administer, a treatment to provide, medical equipment to use, and device options to set in a clinical trial or a pre-clinical trial environment.
Ito, however, teaches “wherein selecting the at least one action comprises selecting at least one of a drug to administer, a treatment to provide, medical equipment to use, and device options to set in a clinical trial or a pre-clinical trial environment” as “The following description will consider the problem of determining an administration action for a clinical trial of a certain drug of a pharmaceutical company. In this case, an action of determining doses of administration to a plurality of subjects and the presence or absence of administration thereto is expressed by a vector a.sub.t having, as components, details of the administration action with respect to each of the subjects. For example, an action of carrying out administration in a dose 1 to a subject A, not carrying out administration with respect to a subject B, and carrying out administration in a dose 2 with respect to a subject C is expressed by a vector a.sub.t=(1,0,2, . . . )” (Paragraph 88).
The examiner further notes that Ito teaches an action of dosages in a clinical trial to administer in a Bandit setting. The combination would result in the bandit setting of Pandey and Guha to also be in the context of clinical trial situations.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Ito’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a bandit solution in differing problems, as noted by Ito (Paragraph 82).
Regarding claim 4, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein selecting the at least one action comprises selecting an experimental option from a plurality of experimental options to evaluate in an experimental environment.
Ito, however, teaches “wherein selecting the at least one action comprises selecting an experimental option from a plurality of experimental options to evaluate in an experimental environment” as “The following description will consider the problem of determining an administration action for a clinical trial of a certain drug of a pharmaceutical company. In this case, an action of determining doses of administration to a plurality of subjects and the presence or absence of administration thereto is expressed by a vector a.sub.t having, as components, details of the administration action with respect to each of the subjects. For example, an action of carrying out administration in a dose 1 to a subject A, not carrying out administration with respect to a subject B, and carrying out administration in a dose 2 with respect to a subject C is expressed by a vector a.sub.t=(1,0,2, . . . )” (Paragraph 88).
The examiner further notes that Ito teaches an action of dosages (i.e. the undefined claimed experimental option in the broadest reasonable interpretation) in a clinical trial (i.e. the undefined experiment in the broadest reasonable interpretation) to administer in a Bandit setting. The combination would result in the bandit setting of Pandey and Guha to also be in the context of clinical trial (i.e. experiment) situations.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Ito’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a bandit solution in differing problems, as noted by Ito (Paragraph 82).
Regarding claim 6, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein selecting the at least one action comprises selecting at least one of a price at which to set one or more products, an amount of the one or more products to order, and a time at which to the order one or more products in a food retail environment.
Ito, however, teaches “wherein selecting the at least one action comprises selecting at least one of a price at which to set one or more products, an amount of the one or more products to order, and a time at which to the order one or more products in a food retail environment” as “The following description will consider the problem of determining the rates of increase/discount on beer prices of individual companies in a certain store. In this case, an action of determining the rates of increase/discount on the beer prices of the individual companies is expressed by a vector a.sub.t having, as components, the rates of increase/discount on the beer prices of the individual companies. For example, an action of setting a beer price of a company A to a fixed price, setting a 20% increase in a beer price of a company B from a fixed price, and setting a 10% reduction in a beer price of a company C from a fixed price is expressed by a vector a.sub.t=(0,+2,−1, . . . ). Then, it is assumed that a loss l.sub.t.sup.Ta.sub.t is obtained as a feedback. In this case, application of the above-described information processing method S1 makes it possible to determine rates of increase/discount that reduce a loss” (Paragraph 86).
The examiner further notes that Ito teaches an action of price determination in a retail environment in a Bandit setting. The combination would result in the bandit setting of Pandey and Guha to also be in the context of price determinations.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Ito’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a bandit solution in differing problems, as noted by Ito (Paragraph 82).
13. Claims 5 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Pandey et al. (Article entitled “Multi-armed Bandit Problems with Dependent Arms”, dated 2007), in view of Jiang et al. (U.S. PGPUB 2022/0070698), and further in view of Guha et al. (Article entitled “Multi-Armed Bandit Problems with Delayed Feedback”, dated 18 June 2013) as applied to claims 1, 7-8, 12-13, 15-18, and 20 above, and further in view of Bonnefoi et al. (Article entitled “Multi-Armed Bandit learning in IoT networks: Learning helps even in non-stationary settings”, dated 02 July 2018).
14. Regarding claim 5, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein selecting the at least one action comprises selecting a channel from a plurality of channels to use in an Internet-of-Things (IoT) environment.
Bonnefoi, however, teaches “wherein selecting the at least one action comprises selecting a channel from a plurality of channels to use in an Internet-of-Things (IoT) environment” as “We evaluate the performance of two classical MAB learning algorithms, UCB1 and Thomson Sampling, to handle the decentralized decision-making of Spectrum Access, applied to IoT networks; as well as learning performance with a growing number of intelligent end-devices” (Abstract), “Considered alone, each dynamic device implements a learning algorithm to play a bandit game, the device is consequently a smart device. In each time slot, if it has to communicate (which happens with probability p), then it chooses a channel and it receives a reward 1 if the transmission is successful, 0 otherwise. Each device aims at maximizing the sum of the rewards collected during its communication instants, which shall indeed maximize the fraction of successful transmissions. Besides the modified time scale (rewards are no longer collected at every time step), this looks like a bandit problem. However, it cannot be modeled as a stochastic MAB, as the rewards are clearly not i.i.d: they not only depend on the (stationary, i.i.d) behavior of the static devices, but also on the behavior of other smart devices, that is not stationary (because of learning)” (Page 8, Section 4.4), and “We can see that when the proportion of end-devices is low (e.g., 1% of devices are dynamic), the optimal policy provides an improvement of 16% compared to random channel selection” (Page 11, Section 5).
The examiner further notes that Bonnefoi teaches an action of IoT channel selection in a Bandit setting. The combination would result in the bandit setting of Pandey and Guha to also be in the context of IoT channel selection.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Bonnefoi’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a method to help fit more devices in IoT networks, as noted by Bonnefoi (Abstract).
Regarding claim 10, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein the bandit score is determined using an infinite time horizon algorithm.
Bonnefoi, however, teaches “wherein the bandit score is determined using an infinite time horizon algorithm” as “We evaluate the performance of two classical MAB learning algorithms, UCB1 and Thomson Sampling, to handle the decentralized decision-making of Spectrum Access, applied to IoT networks; as well as learning performance with a growing number of intelligent end-devices” (Abstract).
The examiner further notes that Bonnefoi teaches the use of UCB1 and Thompson Sampling (i.e. infinite time horizon algorithms). The combination would result in the use of such algorithms in the bandit setting of Pandey and Guha.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Bonnefoi’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a method to help fit more devices in IoT networks, as noted by Bonnefoi (Abstract).
15. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Pandey et al. (Article entitled “Multi-armed Bandit Problems with Dependent Arms”, dated 2007), in view of Jiang et al. (U.S. PGPUB 2022/0070698), and further in view of Guha et al. (Article entitled “Multi-Armed Bandit Problems with Delayed Feedback”, dated 18 June 2013) as applied to claims 1, 7-8, 12-13, 15-18, and 20 above, and further in view of Meshram et al. (Article entitled “On the Whittle Index for restless multi-armed hidden markov bandits”, dated 19 December 2017).
16. Regarding claim 9, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein the bandit score is determined using a Whittle index.
Meshram, however, teaches “wherein the bandit score is determined using a Whittle index” as “Our interest is in a policy to select the arm(s) in each time step that maximizes the infinite horizon discounted reward. Specifically, we seek the use of Whittle’s index in selecting the arms” (Abstract).
The examiner further notes that Meshram teaches the concept of using a whittle index to select arms in a bandit setting. The combination would result in the use of a whittle index to select the arms in the bandit settings of Pandey and Guha.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Meshram’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a method for maximizing infinite-horizon throughput, as noted by Meshram (Page 2, Section 1(B)).
17. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Pandey et al. (Article entitled “Multi-armed Bandit Problems with Dependent Arms”, dated 2007), in view of Jiang et al. (U.S. PGPUB 2022/0070698), and further in view of Guha et al. (Article entitled “Multi-Armed Bandit Problems with Delayed Feedback”, dated 18 June 2013) as applied to claims 1, 7-8, 12-13, 15-18, and 20 above, and further in view of Liu et al. (Article entitled “Risk-Aware Multi-Armed Bandits with Refined Upper Confidence Bounds”, dated 2020).
18. Regarding claim 14, Pandey, Jiang, and Guha do not explicitly teach a method comprising:
A) wherein the rewards measure an uncountable set of outcomes defining a continuous probability distribution.
Liu, however, teaches “wherein the rewards measure an uncountable set of outcomes defining a continuous probability distribution” as “In this paper, we study the risk-aware MAB problem based on the MV paradigm and propose two novel risk-aware MAB algorithms. The key difference of the proposed algorithms from the existing ones (which utilize Hoeffding inequality to drive the confidence bound of the reward variance) is that a finer confidence bound is derived by determining the distribution of the sample variance. Focusing on the MABs with continuous reward distributions, we first build a finer upper confidence bound (UCB) of the MV with a Gaussian reward assumption and design a Gaussian risk aware-upper confidence bound (GRA-UCB) algorithm to solve the risk-aware MAB” (Page 1, Section 1).
The examiner further notes that the instant specification is utterly devoid of any explanation of what the claimed uncountable set of outcomes defining a continuous probability distribution constitutes. Nevertheless, Liu teaches the concept of a Gaussian (i.e. a continuous probability distribution) rewards that are continuous (i.e. uncountable). The combination would result in the use of such a specified bandit setting in the bandit settings of Pandey and Guha.
It would have been obvious to one of ordinary skill in the art before the effective filing date of instant invention to combine the teachings of the cited references because teaching Liu’s would have allowed Pandey’s, Jiang’s, and Guha’s to provide a method for achieving long term learning regret, as noted by Liu (Page 4, Section VI).
Allowable Subject Matter
19. Claims 11 and 19 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 101, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Specifically, although the prior art (See Pandey and Guha) clearly teach determining bandit score, the detailed methodology of the defined discrete configuration process is not found in the prior art, in conjunction with the rest of the limitations of the parent claims.
Conclusion
20. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. PGPUB 2009/0043597 issued to Agarwal et al. 12 February 2009. The subject matter disclosed therein is pertinent to that of claims 1-20 (e.g., methods to use multi-arm bandits).
Article entitled “Bandit-Based Monte Carlo Optimization for Nearest Neighbors”, by Bagaria et al., dated 28 April 2021. The subject matter disclosed therein is pertinent to that of claims 1-20 (e.g., methods to use multi-arm bandits).
Contact Information
21. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Mahesh Dwivedi whose telephone number is (571) 272-2731. The examiner can normally be reached on Monday to Friday 8:20 am – 4:40 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Charles Rones can be reached (571) 272-4085. The fax number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see 20. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Mahesh Dwivedi
Primary Examiner
Art Unit 2168
June 23, 2026
/MAHESH H DWIVEDI/Primary Examiner, Art Unit 2168
Prosecution Insights