Thompson sampling is old. William R. Thompson published the probability-matching idea in “On the Likelihood that One Unknown Probability Exceeds Another” in 1933. BanditBuzz’s free standard policy still uses a Beta-Bernoulli form of that rule. That is not nostalgia. It is a product choice with a short list of reasons, a short list of assumptions, and a short list of failure cases.

This post explains what the running code does: how posteriors form from matured decisions, how one draw per option picks an action, how explore versus exploit labels are assigned, how new options enter, how value mode changes the score, and how 500 Monte Carlo draws become the logged propensity. It also says where the method stops being enough.

Why start here

A decision product that claims “nothing to configure” cannot ask the buyer for an exploration rate, a burn-in schedule, or a training set. Classic multi-armed bandits already have finite-time regret foundations; Auer, Cesa-Bianchi, and Fischer (2002) give that finite-time analysis for stochastic bandits and a useful UCB baseline. BanditBuzz does not ship that UCB rule. It ships Thompson sampling because the rule stays randomized without a separate exploration knob, and because that randomization is what later offline evaluation needs.

Chapelle and Li (2011) compared Thompson sampling with several alternatives and found the Beta-Bernoulli form a strong, simple baseline. In their tests, randomized Thompson sampling also handled delayed batch updates better than deterministic UCB. That matters for BanditBuzz: rewards arrive through a later event, often after identity resolution, and posteriors update on a nightly batch, not after every click.

Gao, Han, Ren, and Zhou (2019) show that bandit learning can keep strong regret rates with a small number of update batches under their assumptions. Their result does not prove that any particular nightly schedule is optimal. It does support treating batch updates as a valid design rather than a broken real-time learner.

None of those papers claims Thompson sampling is universally best. BanditBuzz does not claim that either. The method is a useful first policy when its assumptions fit: roughly independent binary (or value-scaled) outcomes per option, a stable enough reward process between updates, and an option list that matches the real choice.

The posterior the product actually stores

Each option on a decision point keeps three matured counts:

  • matured_decision_count: decisions whose reward window has closed, or that already earned a reward
  • reward_count: how many of those matured decisions earned the reward
  • reward_value_sum: the sum of recorded reward values when value mode is on

Young decisions stay out of those counts. An unrewarded choice is not a failure until its full window closes. That maturation rule is the difference between “we have not heard yet” and “nothing happened.” Without it, a seven-day purchase window would punish every fresh option for rewards that have not had time to land.

The conversion posterior is:

θ ~ Beta(reward_count + 1, matured_decision_count − reward_count + 1)

That is exactly Beta(r + 1, n − r + 1) with a Beta(1, 1) prior. A brand-new option, or an option string the policy has never seen, starts at Beta(1, 1): uniform on [0, 1]. After two rewards in ten matured decisions it is Beta(3, 9). After eight rewards in ten it is Beta(9, 3).

Those three shapes are useful to picture, but they are an explanation, not a dashboard widget. The product stores counts and samples from the resulting Beta. It does not publish density plots for each option.

One decision: draw, pick, label

On each choose call (after the holdout check), the policy does one Thompson round:

  1. For every offered option name, build the Beta parameters from stored counts, or fall back to zeros for an unseen name.
  2. Draw one conversion sample from that Beta.
  3. Optionally scale the sample in value mode (next section).
  4. Pick the option with the largest score.
  5. Compare the winner’s name to the option with the largest posterior mean (α / (α + β)). If they match, label the decision exploit. If they differ, label it explore.

The explore/exploit label is descriptive, not a second policy. The sampler already chose. The label only records whether this draw agreed with the current mean ranking. That helps the scoreboard show mix over time without inventing a separate ε-greedy schedule.

Holdout is separate. About 10% of stable person keys for a decision-point version are assigned by a cryptographic hash of the decision point and person key. Holdout people get nothing when that string is offered (probability 1), otherwise a uniform random offered option. Their selection probabilities are exact. They do not run Thompson sampling. The rest of this post is about the treated path.

New options enter without a registration step

Option names are strings in the request. There is no prior catalog. If a later call adds offer_20_off, the policy has no row for it yet, so scoring uses the zero-count fallback: Beta(1, 1). The new option enters the exploration set automatically. That matches the product rule that options are just strings and new ones should be tried without a configuration ceremony.

The flip side is also true. There are no traffic floors, ceilings, or pinned options on the free tier. A weak new option can still win draws while its posterior is wide. A strong old option can still lose draws. That is the point of probability matching. Buyers who need hard constraints should enforce them outside the learner (by omitting forbidden options) rather than expecting the sampler to respect unstated floors.

Value mode: what it does and what it does not

When the reward block includes a value field, the standard policy does not switch to a full continuous-reward model. It keeps the Beta conversion posterior and changes the score:

score = sampled_conversion × (reward_value_sum + 1) / (reward_count + 1)

The + 1 terms are a crude pseudo-observation so an unseen option is not multiplied by zero. The landing page states the limit plainly: value mode scales sampled conversion by observed mean reward value; it is not yet a fully value-scaled inference path. Live lift on the scoreboard still uses binary reward incidence. Treat value mode as an honest v1 bias toward options that convert and pay more when they convert, not as a claim that the posterior is a proper Bayesian model of dollars.

Why the propensity is estimated with 500 draws

Offline policy evaluation needs the probability that the logging policy would have chosen the logged action. For holdout, that probability is exact. For Thompson sampling, the exact probability is an integral over the joint draw. BanditBuzz does not compute that integral. It estimates it.

The code reruns the same scoring rule used to choose the action, 500 times, and counts how often each offered option wins. Then it applies one-count (Laplace) smoothing:

P̂(option) = (wins + 1) / (500 + number_of_offered_options)

Every offered option gets at least one pseudo-win. No logged Thompson decision can carry probability 0. The probabilities still sum to 1. The landing page is explicit that these are estimates, not exact integrals and not clipped probabilities.

Five hundred draws over a handful of options is cheap enough to run on the decide path. The push sweeper estimates the probability vector once per sweep when posteriors are fixed, then picks per person without re-estimating 500 times for every candidate. That is an implementation detail, not a change in the estimator.

Logged propensities are what make later replay possible. They are also why the free policy’s randomization is a feature: a deterministic greedy rule would put most mass on one action and leave little support for counterfactual evaluation.

Delayed rewards and batch updates

The product does not update posteriors on every decision. Attribution waits for reward windows and identity joins, then a periodic job matures decisions and recounts. Nightly is the operating choice. Between updates, every treated decide call still draws fresh samples from the current posteriors, so traffic keeps exploring even while counts are frozen.

That design leans on the delayed-feedback evidence above: Chapelle and Li’s robustness findings for Thompson sampling under delay in their settings, and Gao et al.’s support for batched updates under theirs. It also inherits the limits. If the world shifts faster than the batch and the full-history posteriors can move, the policy can be slow. Action-choice evidence currently keeps full weight. There is no automatic discount for seasonality or creative wear-out on the free action policy. Timing uses a separate 90-day window; action learning does not.

What this policy is good for

Put the pieces together and the free tier’s job becomes clear:

  • Start without training data: every option begins at Beta(1, 1).
  • Explore without an exploration-rate setting: uncertainty in the posterior drives the draws.
  • Stay randomized between nightly updates: each decide call samples again.
  • Log a propensity for each choice: 500 smoothed Monte Carlo draws on the treated path, exact probabilities on holdout.
  • Accept delayed rewards: only matured decisions train the posterior.
  • Accept new options: unseen names sample from the uninformative prior.

Those are the reasons to put a 1933 method in a new decision product. Age is a benefit only when the assumptions fit. Binary or value-scaled outcomes, a coherent option list including nothing when “whether to act” matters, and a reward process that does not thrash between updates are the fit conditions.

What this policy is not

It is not the paid per-person policy. That upgrade uses hierarchical linear Thompson sampling with a compact context vector and a separate replay preview. The free Beta posteriors ignore person features on purpose.

It is not a claim that option-level reward rates are causal contrasts. The policy itself changes who receives each option, so per-option rates on the scoreboard are descriptive.

It is not a solution to drift. Full-history Beta counts will overweight an old world if the new world differs.

It is not a solution to overlapping actions, interference between people, or long-term pacing. Last-decision-wins attribution is a product rule. Holdout measures the whole live policy against its control, not why a particular template won.

It is not optimal for every reward, every action set, or every traffic pattern. Chapelle and Li’s strong baseline result, Gao et al.’s batch results, and Auer et al.’s finite-time stochastic analysis are evidence for related designs under stated assumptions. They are not a warranty for every BanditBuzz account.

A short walkthrough

Suppose a win-back decision offers 10_percent, free_shipping, and nothing. After some weeks the matured counts might look like:

Option Matured Rewards Posterior
10_percent 10 2 Beta(3, 9)
free_shipping 10 8 Beta(9, 3)
nothing 0 0 Beta(1, 1)

On the next treated decide call the policy draws once from each Beta, picks the largest draw, labels explore or exploit against the mean ranking, and estimates selection probabilities with 500 fresh Monte Carlo rounds under the same scoring rule. If value mode is on, each conversion draw is scaled by that option’s smoothed mean reward value before the argmax. Overnight, newly matured decisions update the counts. Tomorrow’s draws use the new Betas.

That is the whole free policy in production terms. The mathematics is short. The product work sits around it: maturation, holdout, propensity logging, and the honesty to keep the method’s limits visible.