DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Baihan Lin (hereinafter Lin) (“Evolutionary Multi-Armed Bandits with Genetic Thompson Sampling”, 04/26/2022) in view of Djallel Bouneffouf (hereinafter Bouneffouf) (“Toward Optimal Solution for the Context-Attentive Bandit Problem”, 08/01/2021).
Regarding claim 1, Lin teaches;
A method for solving a
([Abstract] we propose the Genetic Thompson Sampling, a bandit algorithm that keeps a population of agents and update them with genetic principles such as elite selection, crossover and mutations)
NOTE: Teaches a method for solving a bandit problem using a genetic, evolution based Thompson sampling algorithm. The evolutionary agents constitute genomes under the broadest reasonable interpretation.
identifying a ([pg. 3] smaller Sf and Ff which makes the stochastic sampling more noisy, encouraging more exploration.) ([pg. 5] we let the agents make decisions for 100 steps and reveal the reward and cost at each step as their feedbacks);
initializing a population of genomes ([pg. 3] M is a population of Thompson Sampling agents … Initialize: M) for use with the exploration parameters ([pg. 3] agents, each with their starting … fitness parameters Sf and Ff)
initializing, for each genome in the population of genomes, exploration parameter values ([pg. 3] agents, each with their starting … fitness parameters Sf and Ff which are both initiated as 1) ([pg. 3] Repeat for each step t … For each agent m ∈ {m ∈ M|am == a∗} … Smf := Smf + r ... Smf := Fmf + (1 −r)), and a cumulative reward for the genome ([pg. 3] R is the reward function … For each agent m ∈ M: Smf = 1 … Observe r ∈ Ra∗ … Smf := Smf + r);
choosing an action arm A(t) for the genome at each iteration; observing a reward R(t)
[pg. 3]
PNG
media_image1.png
211
617
media_image1.png
Greyscale
and updating the cumulative reward for the genome ([pg. 3] for each step t … For each agent m … Smf := Smf + r) over the maximum number of iterations for the genome evaluation for the genome ([pg. 5] we let the agents make decisions for 100 steps);
selecting a subset of the population of genomes ([pg. 4] choosing the top N agents based on the fitness score) based on the cumulative rewards ([pg. 3] Smf := Smf + r) accrued for respective genomes in the population of genomes ([pg. 3] for each step t … For each agent m ∈ {m ∈ M|am == a∗} … Smf := Smf + r … Get fitness score fm ∼ Beta(Smf, Fmf), ∀m ∈ M) over the maximum number of iterations for respective genome evaluations ([pg. 5] we let the agents make decisions for 100 steps);
and replacing one or more of the population of genomes with newly created offspring genomes ([pg. 2] fitter models are then used as “parents”, to be paired up to generate “offspring”. These offspring are introduced into the population of the next generation until it is full … [pg. 4] choosing the top N agents … kept as elites … For each freed out spaces for new children, we randomly select two parents from the elite pool).
Lin fails to teach but Bouneffouf teaches;
A method for solving a contextual bandit problem using an
([Abstract] In this paper, we analyze and extend an online learning framework known as Context-Attentive Bandit, We derive a novel algorithm, called Context-Attentive Thompson Sampling (CATS), which builds upon the Linear Thompson Sampling approach, adapting it to Context-Attentive Bandit setting.)
NOTE: Teaches solving contextual bandit problem using a linear Thompson sampling algorithm (CATS).
(Reasoning as to why it would be obvious for CATS to be incorporated into the evolutionary algorithm of Lin will be explained later below)
identifying a contextual bandit problem having exploration parameters ([pg. 3495] the distribution parameter α>0 used in linear Thompson Sampling)
NOTE: α represents the distribution parameter of the algorithm.
[pg. 3496]
PNG
media_image2.png
54
576
media_image2.png
Greyscale
NOTE: Algorithm 2 shows that for each arm k, CATS samples ˜µk (where µk represents an unknown weight factor for arm k, influencing how likely it is for an action / arm to be selected, though the sampled version ˜µk is the value used in the algorithm) from a gaussian centered at ^µk (the predicted µk) with a scaled covariance
PNG
media_image3.png
26
56
media_image3.png
Greyscale
. The scaled covariance (which is scaled using α) indicates how far from ^µk to sample, i.e. how much to explore. Thus, α can be considered an exploration parameter of the contextual bandit problem, as it influences how much the model explores.
and feature subsets ([pg. 3495] features coming from CV+U are used, where CV is a fixed subset of C, and CU is any subset of C\CV);
initializing a
[pg. 3496]
PNG
media_image4.png
123
451
media_image4.png
Greyscale
[pg. 3496]
PNG
media_image5.png
80
401
media_image5.png
Greyscale
NOTE: Teaches the initialized CATS algorithm for use with the exploration parameters (α) and the feature subsets.
initializing a random feature subset to use at each iteration
[pg. 3496]
PNG
media_image6.png
156
450
media_image6.png
Greyscale
calculating,
[pg. 3496]
[AltContent: rect][AltContent: rect]
PNG
media_image7.png
133
596
media_image7.png
Greyscale
NOTE: Discloses the expected reward formula.
[pg. 3496]
[AltContent: textbox (1)][AltContent: rect]
PNG
media_image8.png
85
427
media_image8.png
Greyscale
NOTE: Teaches calculating an expected reward using the exploration parameters (box 1 pictures the expected value formula, utilizing the aforementioned ˜µk, which is derived using the aforementioned exploration parameter, α) and the feature subsets (c^V+U(t)).
OBVIOUSNESS TO COMBINE LIN AND BOUNEFFOUF:
Lin and Bouneffouf are analogous art to each other and to the present disclosure as they both pertain to bandit algorithms and Thompson sampling. Lin keeps a population of Thompson sampling bandits and updates the population via selection, crossover, and mutation, while Bouneffouf teaches a Context Attentive bandit problem and a Context Attentive Thompson Sampling algorithm, which applies Linear Thompson Sampling in the contextual bandit setting, as well as contextual Linear Thompson Sampling implementations employing randomly selected feature subsets. Additionally, Lin indicates that incorporating the disclosed genetic algorithm into the bandit problem significantly improves bandit performance in nonstationary settings ([Lin, Abstact] Empirical results in multi armed bandit simulation environments and a practical epidemic control problem suggest that by incorporating the genetic algorithm into the bandit algorithm, our method significantly outperforms the baselines in nonstationary settings) and also indicates that it would be reasonable to consider applying their evolutionary bandit framework to contextual bandits in future works ([Lin, pg. 7] Future work include extending this evolutionary bandit framework to contextual bandits). Additionally, Bouneffouf indicates that their method for solving a contextual bandit problem using a linear Thompson sampling algorithm outperforms the state of the art in majority of cases. (“[Bouneffouf, pg. 3496] our methods outperforming the state of the art in the majority of cases”). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Lin’s evolutionary Thompson-sampling framework such that the respective Thompson-sampling agents implement Bouneffouf’s contextual Linear Thompson Sampling procedure. Lin expressly suggests extending its evolutionary bandit framework to contextual bandits, and Bouneffouf provides a known contextual Thompson-sampling implementation. Such a combination would predictably permit Lin’s population-based genetic selection, crossover, and mutation to be applied to contextual Thompson-sampling agents while retaining Bouneffouf’s contextual feature-based arm selection.
In the context of this combination, Lin’s respective agents/genomes would each implement an instance of Bouneffouf’s contextual Linear Thompson Sampling procedure, with Lin’s genetic operations applied across the resulting population. Accordingly, the combination teaches;
A method for solving a contextual bandit problem (Bouneffouf) using an Evolution (Lin) Linear Thompson Sampling (Bouneffouf) (ELINTS) algorithm, the method comprising: identifying a contextual bandit problem having exploration parameters and feature subsets (Bouneffouf) and a maximum number of iterations for each genome evaluation (Lin); initializing a population of genomes for use with the exploration parameters (Lin) and the feature subsets (Bouneffouf); initializing, for each genome in the population of genomes (Lin), exploration parameter values, a random subset of features to use at each iteration (Bouneffouf), and a cumulative reward for the genome (Lin); … for each genome in the population of genomes and for each iteration of the maximum number of iterations for a genome evaluation for the genome (Lin), [calculating] an expected reward using the exploration parameters values and the random subset of features for the genome (Bouneffouf);
Regarding claim 2, Lin in view of Bouneffouf teaches;
The method of claim 1
(Using the same reasoning as in claim 1)
Lin teaches;
wherein the ([pg. 6] selecting a proper … number of arms), a total number of generations, (Note: new generation created each step … [pg. 3] each agent m ∈ M … Repeat for each step t … Selection: Mp = selection(M,f). Crossover: Mc = crossover(Mp). Mutation: M = mutation(Mc) … [pg. 5] we let the agents make decisions for 100 steps) a population size ([pg. 6] selecting a proper population size), a probability of mutation ([pg. 4] We first set a mutation rate to indicate how many mutations we want to have in this model. The higher the number, the more mutations the model will introduce) and a maximum number of iterations for each genome evaluation ([pg. 5] we let the agents make decisions for 100 steps).
Lin fails to teach but Bouneffouf teaches;
wherein the contextual bandit problem includes a plurality of input parameters, wherein the plurality of input parameters include … a total number of features, ([pg. 3494] The algorithm takes the total number of features N).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 3, Lin in view of Bouneffouf teaches;
The method of claim 1
(Using the same reasoning as in claim 1)
Lin teaches;
wherein initializing a population of genomes includes iterating through a plurality of generations and for each generation, identifying the population of genomes.
PNG
media_image9.png
659
533
media_image9.png
Greyscale
NOTE: Teaches initializing a population of genomes including iterating through a plurality of generations (for each step/generation t) and for each generation, identifying the population of genomes (at teach step/generation t, the population of genomes M is identified using selection, crossover, and mutation).
Regarding claim 4, Lin in view of Bouneffouf teaches;
The method of claim 1
(Using the same reasoning as in claim 1)
Lin teaches;
wherein initializing the population of genomes includes, for each genome in the population of genomes, initializing the exploration parameters values ([pg. 3] agents, each with their starting … fitness parameters Sf and Ff which are both initiated as 1) ([pg. 3] Repeat for each step t … For each agent m ∈ {m ∈ M|am == a∗} … Smf := Smf + r ... Smf := Fmf + (1 −r)).
Lin fails to teach but Bouneffouf teaches;
initializing … the random feature subset
[pg. 3496]
PNG
media_image6.png
156
450
media_image6.png
Greyscale
OBVIOUSNESS: Using the same reasoning from claim 1
Regarding claim 5, Lin in view of Bouneffouf teaches;
The method of claim 1
(Using the same reasoning as in claim 1)
Lin teaches
genome … each iteration of the genome evaluation for the genome (using the same reasoning from claim 1)
Lin fails to teach but Bouneffouf teaches;
wherein calculating the expected reward includes calculating the expected reward for an arm K
[pg. 3496]
PNG
media_image10.png
22
104
media_image10.png
Greyscale
NOTE: Teaches wherein calculating an expected reward includes calculating the expected reward for an arm K (calculating the aforementioned expected reward for arm k,
PNG
media_image10.png
22
104
media_image10.png
Greyscale
) for each iteration (iteration t) and identifying an arm K with the highest expected reward (selecting the arm that satisfies the argmax, i.e. having the highest expected reward).
OBVIOUSNESS: Using the same reasoning from claim 1
Regarding claim 6, Lin in view of Bouneffouf teaches;
The method of claim 5
(Using the same reasoning as in claim 5)
Lin teaches
choosing an action arm A(t) … for the genome (using the same reasoning from claim 1)
Lin fails to teach but Bouneffouf teaches;
wherein choosing an action arm A(t) includes selecting,
[pg. 3496]
PNG
media_image11.png
18
347
media_image11.png
Greyscale
NOTE: Teaches wherein choosing an action arm (select arm k(t)) includes selecting an action arm by selecting the arm K having the highest expected reward (k(t) is selected by having the highest expected reward, as previously mentioned in claim 5).
OBVIOUSNESS: Using the same reasoning from claim 1
Regarding claim 7, Lin in view of Bouneffouf teaches;
The method of claim 6
(Using the same reasoning as in claim 6)
Lin teaches;
and updating a cumulative reward for the genome over the maximum number of iterations for the genome evaluation for the genome ([pg. 3] Repeat for each step t … For each agent m ∈ {m ∈ M|am == a∗} … Smf := Smf + r … [pg. 5] we let the agents make decisions for 100 steps).
Lin fails to teach but Bouneffouf teaches;
wherein observing a reward R(t) includes observing the reward R(t) for the selected action arm A(t)
[pg. 3496]
PNG
media_image12.png
39
399
media_image12.png
Greyscale
OBVIOUSNESS:
Using the same reasoning as in claim 1.
Regarding claim 8, Lin in view of Bouneffouf teaches;
The method of claim 7
(Using the same reasoning as in claim 7)
Lin teaches;
wherein observing a reward R(t) further includes updating the exploration parameter values for the genome based on the observed reward R(t) using at least one of a mutation algorithm and a crossover algorithm.
[pg. 3]
PNG
media_image13.png
494
521
media_image13.png
Greyscale
NOTE: Teaches observing a reward further including updating exploration values (the aforementioned exploration values Sf^m and Ff^m) based on the observed reward (Sf^m and Ff^m are updated using the observed reward r). In a given step t, the population of agents M passes through a mutation and crossover algorithm, resulting in a new population. In the following step, the reward is observed from the actions of the new population, thus teaching using at least one of a mutation algorithm and a crossover algorithm in the process of updating the exploration values based on the observed reward.
Regarding claim 9, Lin in view of Bouneffouf teaches;
The method of claim 5
(Using the same reasoning as in claim 5)
Lin teaches;
wherein selecting a subset of the population of genomes includes selecting the subset of the population of genomes using a genetic selection approach.
([pg. 2] For each generation t, a fitness score is computed for each candidate model m ∈ Mt, and only a subset of the candidate models, deemed as elites fit enough by the fitness scores, are kept in the population of the next generation. The not-so-fit models which doesn’t match the criterion are eliminated, making room for new individuals.)
Regarding claim 10, Lin in view of Bouneffouf teaches;
The method of claim 1
(Using the same reasoning as in claim 1)
Lin teaches;
wherein replacing one or more of the population of genomes includes creating the newly created offspring genomes by applying a crossover algorithm and a mutation algorithm on the selected subset of existing genomes.
[pg. 3]
PNG
media_image14.png
124
444
media_image14.png
Greyscale
Regarding claim 11;
Claim 11 is a system claim directly corresponding to method claim 1, with one different limitation, which is taught by Lin in view of Bouneffouf;
Lin teaches;
a machine learning system ([pg. 7] we propose a hybrid online learning framework…)
(the remaining limitations are taught using the same reasoning as in claim 1)
Regarding claims 12 – 19,
Claims 12 through 19 are system claims directly corresponding to method claims 2-9, respectively, and are therefore rejected using the same reasoning
Regarding claim 20,
Claim 20 is a CRM claim directly corresponding to method claim 1, and is therefore rejected using the same reasoning.
Response to Arguments
The claim objections from the previous office action have been withdrawn in view of the amendments.
Applicant's arguments, filed 07/15/2026, regarding the rejection of claims 1-20 under 35 U.S.C. & 103 have been fully considered but they are not persuasive. Starting on page 2, the acclicant contends that “Applicant respectfully submits that the cited references, whether considered alone or in combination, do not disclose or suggest the claimed same-genome evaluation sequence of independent claims 1, 11, and 20 as amended, and therefore respectfully further submits that the rejection under § 103 is overcome … Applicant has amended independent claims 1, 11, and 20 to recite, among other features, "a maximum number of iterations for each genome evaluation," initializing "for each genome in the population of genomes, exploration parameter values, a random subset of features to use at each iteration, and a cumulative reward for the genome," calculating "for each genome in the population of genomes and for each iteration of the maximum number of iterations for a genome evaluation for the genome, an expected reward using the exploration parameter values and the random subset of features for the genome," updating "the cumulative reward for the genome over the maximum number of iterations for the genome evaluation for the genome," and selecting "a subset of the population of genomes based on the cumulative rewards accrued for respective genomes in the population of genomes over the maximum number of iterations for respective genome evaluations. Lin, inter alia, is silent as to "initializing, for each genome in the population of genomes, exploration parameter values, a random subset of features to use at each iteration, and a cumulative reward for the genome." Lin is also silent as to "calculating, for each genome in the population of genomes and for each iteration of the maximum number of iterations for a genome evaluation for the genome, an expected reward using the exploration parameter values and the random subset of features for the genome." Lin is further silent as to "updating the cumulative reward for the genome over the maximum number of iterations for the genome evaluation for the genome," and is silent as to "selecting a subset of the population of genomes based on the cumulative rewards accrued for respective genomes in the population of genomes over the maximum number of iterations for respective genome evaluations." Furthermore, Bouneffouf does not remedy these deficiencies of Lin." Examiner respectfully disagree. As reflected by the present office action, Lin and Bouneffouf, in combination, suggest or teach the recited limitations. In particular, Lin’s algorithm 4 performs relevant operations separately for each indexed agent / genome m in the population, M. For each agent, Lin teaches initializing exploration related parameter values (Smf and Fmf) and a cumulative reward for each genome (Smf), updating the cumulative reward Smf for the genome over the maximum number of iterations (100 steps) for the genome evaluation for the genome, and selecting a subset of the population of genomes based on the cumulative rewards Smf accrued for respective genomes m in the population of genomes M over the maximum number of iterations (100 steps) for respective genome evaluations. Bouneffouf further teaches initializing a random subset of features for use by a bandit problem implementing contextual linear Thompson sampling and calculating an expected reward for arm selection using an exploration parameter value and the random subset of features. The office action further provides a motivation to modify Lin’s evolutionary Thompson-sampling framework such that the respective Thompson-sampling agents implement Bouneffouf’s contextual Linear Thompson Sampling procedure. In such a combination, Bouneffoufs calculation of the expected reward (as well as other operations in the method of Bouneffouf) would be performed for each iteration of the genome evaluation for each respective genome/agent of Lin that implements Bouneffouf’s contextual linear Thompson sampling. Accordingly, claims 1, 11, and 20, as well as dependent claims 2-10, 12-19, stand rejected under 35 U.S.C. & 103.
CONCLUSION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights