A growth lead opens a lift dashboard and wants a clear answer: did the adaptive policy add value? An analyst has to defend that answer in a review. The hard part is not drawing a chart. The hard part is naming the estimand, keeping assignment honest while the policy keeps learning, and reporting uncertainty in a form that still makes sense when people look at the same board every day.
BanditBuzz answers one narrow question on the live scoreboard. It compares the complete learned policy with a stable randomized holdout on the reward you declared for that decision point. It does not compare with last year. It does not prove that one offered action beat your current journey. It does not promise proof by day 30. The estimate and its range tell you what the current evidence supports, and you decide when the range is narrow enough for the decision at hand.
What the scoreboard is estimating
The live action lift estimand is the difference in matured reward rates:
absolute effect = matured reward rate under the learned policy − matured reward rate in holdout
When the holdout rate is above zero, the scoreboard also reports relative lift as that absolute difference divided by the holdout rate. Both numbers describe the whole policy against its control choice. They are not effects for any single option.
That matters because an adaptive policy changes who receives each option over time. A later section covers why per-option reward rates remain useful summaries and still fail as option effects. The holdout comparison is the causal claim the product is willing to make on live traffic.
Push decision points require a nothing option, so holdout is no action. For a request-time decision without nothing, holdout receives a uniform random offered option. In both cases the treated side is everyone who was not assigned to holdout and whose decision has matured. Treated includes explore and exploit choices. The scoreboard is asking whether the complete selection rule changed the declared result relative to the control rule, not whether the latest favorite template is “winning.”
Deterministic assignment from a stable person key
Holdout is not a weekly sample of traffic and not a coin flip on each call. About 10% of stable person keys enter holdout for each decision-point version. Assignment uses SHA-256 of the version ID and the person’s stable key. The first 32 bits of the digest, scaled to the unit interval, decide holdout membership against the 0.1 threshold.
The stable key prefers normalized email, then normalized phone, then a hash of all supplied identifiers. A caller who sends nothing usable cannot be assigned stably, so that person always receives the learned policy. The same key is used for request-time /decide calls and for push sweeps, so a person stays on one side of the experiment for that version whether the product or the client initiated the choice.
Determinism is the point. Replays, retries, and repeated eligibility cycles must not flip people in and out of control. A person who is holdout today under the same version and the same stable key is holdout tomorrow. Tests check that assignment is stable and that the realized fraction sits near the configured 10%.
Holdout choices are logged like any other decision. When nothing is offered, the control choice is nothing with selection probability 1. When nothing is absent, the control choice is uniform over the offered list, with probability 1 / n. Those exact propensities sit in the decision log beside the Monte Carlo estimates used for Thompson picks.
Scope is one decision-point version
Holdout lives inside one decision-point version, not across the whole account and not across every decision name forever.
Changing the reward event, the reward window, or the value field creates a fresh version with a fresh policy and a fresh holdout assignment. The hash includes the version ID, so the same person can land on a different side after a reward change. Existing decisions stay tied to the rule they were made under. Different decision points assign independently: being holdout on winback_email says nothing about upsell_sms.
That design keeps the experiment matched to the result definition. If you redefine what “winning” means, you should not keep pooling old and new rewards under one control assignment. It also creates a product limit you should know: BanditBuzz does not yet stop the same person from entering two decision points, receiving two actions, or affecting both scoreboards. Cross-decision interference is out of scope for the current live range.
A second, related limit is identifier drift. A person who later arrives under a different identifier can move sides if the stable key changes. Prefer the same strong identifier set on decide and event calls so assignment and attribution stay aligned.
The unit is a matured decision
The scoreboard counts matured decisions, not unique people and not open reward windows.
A decision matures when a matching reward event arrives or when the declared reward window closes empty. Young decisions stay out of the lift calculation until that happens. That rule keeps pending choices from looking like failures and keeps the treated and holdout rates on the same footing. Attribution uses last-decision-wins inside the window for the resolved person. That is a product rule. It does not prove that the later action caused the result when actions overlap.
Because the unit is a decision, one person can contribute more than one matured row. Cooldown and push sweeps make that common. The current confidence sequence treats those rows as the observation units. It does not yet cluster by person or otherwise adjust for within-person dependence. The landing page states this limit in plain terms, and the data science brief lists person-clustered inference as an open review question. Until that changes, read the range as decision-level evidence, not as a person-clustered interval.
Absolute effect, relative lift, and what they are not
Once both sides have matured decisions, the analysis layer builds two Bernoulli groups: treated rewards over treated matured count, and holdout rewards over holdout matured count. The absolute difference is their rate gap. Relative lift is that gap divided by the holdout rate when the holdout rate is positive.
Read both. Absolute effect answers “how many percentage points of the declared result moved?” Relative lift answers “how large is that move compared with the control baseline?” A 2-point gain on a 4% holdout rate is a 50% relative lift. The same 2-point gain on a 20% holdout rate is a 10% relative lift. Neither number is more honest than the other; they answer different questions.
Neither number is an option effect. The scoreboard still shows a matured reward rate for each option because operators need volume and conversion context. Those rates are descriptive. Because the policy changes who receives each option, comparing option rates does not identify a fixed causal effect between options. Li, Chu, Langford, and Schapire (2010) show how randomized logging supports offline evaluation of whole policies; their setting is also a reminder that adaptive assignment breaks naive arm-to-arm comparisons. BanditBuzz keeps per-option rows for operations and keeps the holdout comparison for the live causal claim.
A nested timing control, when timing is enabled, answers a different question again: learned window versus random allowed window among non-holdout actions that a direct destination accepted. That is timing lift conditional on acting. It is not whole-policy lift. Do not mix the two when you brief a stakeholder.
Why a confidence sequence instead of a fixed-end interval
People will open the scoreboard before any pre-registered end date. A classical fixed-sample interval assumes one look at a planned sample size. Peeking every morning without adjusting the procedure inflates false confidence under that assumption.
Anytime-valid confidence sequences are built for repeated inspection. Howard, Ramdas, McAuliffe, and Sekhon (2021) develop time-uniform, nonparametric confidence sequences with coverage that remains valid under continuous monitoring under the method’s sampling assumptions. Maharaj et al. (2023) describe asymptotic confidence sequences in an enterprise A/B testing platform, including practical tuning for difference and relative lift reporting.
BanditBuzz uses that production-facing asymptotic form for the live reward-rate difference. The confidence level is 95%. The boundary uses Maharaj et al.’s equation with the production tuning value ρ² = 0.001. The same sequential method labels the scoreboard’s inference_method as asymptotic_confidence_sequence. The range appears only when both sides have observed variation; all-zero or all-one samples do not invent a width.
That choice matches how buyers actually use the product. There is no honest promise of proof in 30 days. Traffic volume, the baseline reward rate, the size of the true gain, and the reward window jointly set how long the range stays wide. Use the point estimate with the range. Wait until the interval is narrow enough, and on the correct side of zero, for the business decision you face. Do not treat a single day’s look as a calendar milestone.
Coverage under repeated people, estimated Thompson propensities elsewhere in the system, and nightly policy refits still needs a public simulation benchmark. The live holdout itself is randomized and deterministic, which is the right foundation. The sequential range is the right reporting shape for a dashboard. It is not a certificate that every product-path dependence has been settled.
Reading the scoreboard
A compact way to hold the same idea in your head:
holdout matured rate ████░░░░░░ 4.0%
treated matured rate ██████░░░░ 6.0%
absolute effect +2.0 points
relative lift +50% vs holdout
95% confidence sequence [ +0.6 , +3.4 ] ← inspect any day; wait for width
The numbers above are schematic. Your board will show the account’s own matured counts. What should stay fixed in the reading order is: name the estimand, confirm both sides have matured volume, read absolute and relative effects together, then judge the sequence’s width and location.
What this design deliberately does not claim
A stable holdout estimates the effect of the full live policy against its control choice. It does not isolate why the policy won. It does not separate exploration cost from exploitation gains inside the treated arm. It does not adjust for interference when two decision points act on the same person. It does not yet report a person-clustered range when one person contributes many matured decisions.
Per-option rates remain on the board because operators need them. They are not a substitute for the holdout comparison. Replay of a candidate contextual policy is a separate estimand against the free logging policy, not against holdout. Timing lift is a third estimand, conditional on acting.
The research links above are the primary sources for the sequential range and for the broader point that randomized logs support policy evaluation. BanditBuzz’s implementation choices (about 10% SHA-256 holdout, version-scoped assignment, matured decision units, 95% level, Maharaj production ρ²) are product settings grounded in that literature, not a reprint of any one paper’s exact experiment.
Practical checklist for growth and analytics
- State the estimand first. Whole-policy matured reward rate versus holdout control on the declared reward.
- Confirm the control choice. Push holdout is
nothing. Request-time holdout withoutnothingis uniform random among offered options. - Stay inside one version. A reward definition change resets assignment. Do not stitch versions into one “campaign lift” without saying so.
- Wait for maturation. Open windows are not zeros.
- Report absolute and relative effects together, with the 95% confidence sequence.
- Treat option rates as descriptive. Do not brief them as winners of a fixed A/B among creatives.
- Name the repeated-person limit when one person can appear many times on the board.
- Let the range, not the calendar, decide when the evidence is strong enough.
If you can defend those eight points, the scoreboard is doing its job: turning live adaptive traffic into a lift claim an analyst can stand behind, without pretending the policy stopped learning so a classical fixed test could finish on schedule.