If you only keep “who got a reward,” you cannot check a new policy against old traffic. Offline evaluation needs the randomization record from the moment of choice: which actions were legal, which one ran, how likely that pick was under the logging policy, who the person was, what context was known then, and which reward rule applied. A conversion table is an outcome table. It is not a decision log.
That distinction matters for anyone building an adaptive system, whether or not you use BanditBuzz. The rest of this post is a concrete checklist of fields, why each one exists, and how BanditBuzz implements the hard parts today, including estimated Thompson propensities and the absence of a full buyer export.
Why rewards alone fail
A common warehouse shape looks safe: person, campaign, creative, purchase flag, timestamp. Analysts then ask, “What if we had sent the other offer?” That question needs counterfactual evaluation. Without the logging probabilities and the offered set, the answer is usually a story about correlation among the people who already received each option.
Li, Chu, Langford, and Schapire (2010) put the problem bluntly in their offline-evaluation section: payoffs are observed only for the arms the logging policy chose, so it is not clear how to score a different policy from that log alone. Their contribution was not “log more columns for curiosity.” It was a method for evaluating explore/exploit strategies from previously recorded randomized traffic when interactions can be treated as i.i.d. across people.
Inverse propensity methods go further: they reweight each observed reward by how often the candidate policy would have chosen the logged action relative to how often the logging policy chose it. Dudík, Langford, and Li (2011) combine that idea with a reward model in the doubly robust estimator. They note that inverse propensity scoring “requires an accurate model of the past policy,” and that their estimator stays accurate when either the reward model or the past-policy model is good. In practice that “past policy model” is usually the propensity you stored when the action was taken. Guessing it later from coarse campaign labels is how bias creeps back in.
Public benchmarks make the same point operational. The Open Bandit Dataset and Pipeline release logged fashion-recommendation traffic with known behavior policies so researchers can compare off-policy estimators against on-policy ground truth. Their abstract frames the whole project around estimating “the performance of hypothetical policies using data generated by a different policy.” If the log does not carry action probabilities and related decision metadata, that research program cannot even start.
Logging alone does not make an estimate valid. You still need overlap, enough effective sample size, a clear estimand, and honest uncertainty. But without the right fields, you never get the chance to be honest.
A logged-row checklist
Treat each decision as a row that a stranger could replay years later. Below is the minimum set BanditBuzz treats as load-bearing, with the reason for each field.
1. Decision-point version and reward rule
Store which decision definition produced the row: the named choice (for example winback_email), the reward event name, the reward window in whole days, and whether a value field is in play. Changing the reward event, window, or value field creates a new decision-point version with a fresh policy and holdout assignment. Old rows stay tied to the rule they were made under. Replay that mixes versions silently answers a question nobody asked.
2. Offered action set
Store the exact option list that was legal on that call (offered_options in BanditBuzz). Candidate policies must be scored only over that set. If Tuesday’s call offered 10_percent, free_shipping, and nothing, and Wednesday’s call dropped free_shipping, Wednesday’s policy probability for free_shipping is undefined for Tuesday’s people. BanditBuzz keeps each row’s offered set as it was; adding or removing an option does not rewrite history and does not by itself create a new version.
3. Chosen action
Store the option that actually ran (option_name). This is the arm whose reward you later observe. Without it you have context and probabilities but no treatment.
4. Logging propensity for the chosen action
Store the probability the logging policy assigned to the chosen action given the offered set and the state used at decision time (selection_probability). This is the denominator in importance weights. Holdout rows get exact probabilities. Thompson rows get a Monte Carlo estimate (details below). A missing or zero propensity on a taken action breaks inverse propensity and doubly robust estimators.
5. Policy label and policy kind
Store how the row was produced: explore, exploit, or holdout for the action policy, plus whether the pick came from the standard or contextual policy (policy and policy_kind). Replay and live scoreboards treat holdout differently from treated traffic. Knowing which learner made the pick also keeps fallback rows honest when context lookup fails and the system drops back to the standard rule.
6. Stable person key and identifiers
Store the identifiers supplied on the call and enough normalized data to reproduce the stable key used for holdout assignment. BanditBuzz stores the identifiers, then derives the key when needed: normalized email, then normalized phone, then a hash of all supplied identifiers. Holdout membership is SHA-256 of the decision-point version row ID and that derived key. Without a reproducible stable key, assignment drifts when the same person returns under a different handle, and person-level cross-fitting for replay becomes guesswork.
7. Context snapshot as of decision time
Store the person features the contextual policy actually saw, or a clear null if none were available. BanditBuzz writes a context_snapshot even for many standard decisions so a later upgrade preview never evaluates old rows with context learned after their rewards arrived. Context that is refreshed at analysis time is not the context that caused the choice.
8. Decision time and reward lifecycle times
Store when the choice was made. For scheduled push, also store intended release time (act_at), when the reward window started (reward_window_started_at), cancellation time if the plan died before release, attribution time, and maturity time. Pending, failed, and canceled actions are not negative training examples; their reward windows start only after successful release for direct destinations. Those timestamps keep delayed feedback from being mistaken for missing feedback.
9. Timing offered set, pick, and propensity (when timing applies)
If the system also chooses a local send window, store the offered timing slots, the selected slot, the timing propensity, the timing policy label, and whether a fallback path was used. Action selection and timing selection are factorized in BanditBuzz. Each side gets its own offered set and propensity. Current previews evaluate them separately: action replay leaves the logged timing process in place, while timing replay conditions on an accepted action. There is no joint action-by-time replay that multiplies both propensities today.
10. Idempotency key
Store a caller-supplied idempotency key when present. Retries must return the same decision rather than invent a second randomized draw. BanditBuzz enforces uniqueness per account for non-null keys on both decisions and events. Without this, duplicate network retries inflate exploration and poison propensity denominators.
11. Reward linkage fields
Eventually attach the attributed event, reward value, and maturity marker to the same decision row. Maturity means the window closed or a reward landed. Training should ignore open windows. Last-eligible-decision attribution is a product rule with limits; overlapping actions for one person still interfere. The point of logging the link on the decision is to keep the treatment and the outcome on one evaluable unit.
If you are designing your own ledger, start with items 2 through 7 on every real-time choose call. Add timing and release fields when you schedule. Add idempotency before you put the call on a hot path.
Changing option sets without rewriting the past
Product teams add and retire offers constantly. The logging mistake is to project today’s catalog backward onto yesterday’s rows, or to delete retired arms from historical offered sets.
Replay must ask: given the options that were legal then, what would the candidate have done? BanditBuzz scores candidates only over each row’s stored offered set. History remains frozen. That keeps importance weights defined and keeps “new option” behavior local to the calls that actually offered it. New options enter exploration with an uninformative prior on those calls; they do not invent probabilities on rows that never listed them.
Exact versus estimated propensities
Not every propensity is the same object.
Holdout assignment is deterministic given the version and person key. When nothing is offered, holdout people receive it with probability 1. Otherwise holdout receives a uniform random option with probability 1 / n. Those numbers are exact.
Thompson sampling has no simple closed form for the probability that a given arm wins the draw. BanditBuzz’s standard fix, in policy.ts, is to estimate the selection probability at decide time with 500 fresh Monte Carlo draws from the same scoring rule used to choose the action, then apply one-count (Laplace) smoothing: (wins + 1) / (500 + option_count). The landing page states the product truth directly: these are estimates, not exact integrals and not clipped probabilities. Timing uses the same style of estimate for its window arms when Thompson chooses the slot.
Why smooth? A zero propensity on a taken action makes every inverse propensity weight undefined or infinite. One extra win per option keeps every probability positive while the vector still sums to one. The estimate is stamped synchronously into the decision insert. There is no later backfill that tries to reconstruct the posterior after more rewards arrive. If you re-estimate tomorrow from tomorrow’s posteriors, you are not logging the policy that acted.
Doubly robust replay still divides by that logged number. BanditBuzz’s preview builds importance weights as candidate probability over logged logging probability, rejects runs when effective sample size is too small or any weight exceeds a launch gate, and refuses to treat a positive point estimate as enough. Estimated propensities add noise and some bias relative to the ideal exact integral. That is a known limit of the running system, called out in the research brief. It is still better than inventing probabilities from marketing labels after the fact.
Context must freeze at choice time
A contextual candidate policy is a function of features. If you rebuild features with later purchases, later clusters, or later model scores, you evaluate a policy that could not have run at the original decision. Snapshotting at decide time is the boring fix.
BanditBuzz stores the snapshot used for the choice. If lookup fails or no context exists, the call falls back to the standard policy and records that path. Replay that requires context simply skips rows without it when computing the contextual estimate, and reports coverage. That is less glamorous than filling missing features from a warehouse join, and more honest.
Action and timing are two logged choices
Scheduled push often needs two answers: which approved action, and when inside the allowed local windows. BanditBuzz logs both. The live global holdout still measures the whole push policy against no action. A separate timing control measures learned timing against random allowed timing, conditional on acting.
Because the propensities are separate, you can evaluate one change without pretending you have identified every action-by-time interaction. The product does not currently ship a joint estimator that multiplies action and timing weights. If your own system needs that estimand, you must design the logging and the estimator for it on purpose, not assume factorization falls out of two columns.
Idempotency and audit limits
Idempotency protects the randomization. It does not create a full audit product.
Today BanditBuzz’s buyer surfaces are the scoreboard and pick destinations. They do not provide a full decision, propensity, context, and reward export, and they do not ship a supported replay package you can download and retrain elsewhere. There is also no buyer-visible change audit log for who edited decision definitions. Keys can be rotated; fine-grained roles and scoped rights are not there yet.
Those gaps matter if your compliance or science team expects to pull raw evaluation logs into an external notebook. Internally the ledger exists and powers the in-product preview. Externally, plan for that missing export until it ships. The Open Bandit work is a reminder that reproducible off-policy research needs both the log schema and a way to move the data into an evaluation pipeline. Product logging without export is only half of that story.
What BanditBuzz does with the log
On each /decide (and on each push sweep pick), the service writes the offered options, choice, propensity, policy labels, person fields, optional context snapshot, and idempotency key in one insert. Attribution later attaches rewards under the declared window. The contextual upgrade preview loads matured rows with known logging probabilities and decision-time context, fits five person-level folds, and applies the doubly robust form from Dudík, Langford, and Li (2011). Action replay uses the logged action propensity. Timing preview uses the logged timing propensity on released, matured actions.
None of that rescues a row that never stored a propensity. That is why propensity logging shipped early in the product sequence: history only becomes replayable from the day you start writing it.
Design rules that travel
Whether you buy a decisioning API or build your own:
- Log the offered set and the propensity at choice time, not at report time.
- Freeze context at choice time.
- Keep reward rules versioned with the decisions they governed.
- Separate exact randomization (holdout, uniform timing control) from estimated Thompson probabilities, and say which is which.
- Do not treat a reward warehouse as a substitute for a decision ledger.
- Do not claim validity from logging alone. Check overlap, effective sample size, and the estimand you actually identified.
Li et al. argued that bandit algorithms can be evaluated from randomized logs when the assumptions hold. Dudík et al. showed how to combine a reward model with those logged probabilities. Open Bandit Dataset showed what a public, propensity-aware log looks like when the industry wants reproducible off-policy work. Your ledger should make those papers usable, not force the next analyst to reverse-engineer a policy from a conversion table.
The decision log is part of the model because the next model will be judged on it. Write the fields as if that judgment is the point.