Product builders keep asking a fair question: if a language model can write copy, summarize customers, and draft API calls, can it also pick the next action?

The short answer from the evidence we trust is narrower. A language model can help you name options, explain results, and draft a decision declaration. Current tests do not support treating it as a drop-in replacement for a randomized policy that must keep exploring while it exploits.

That is not a claim that language models are useless for decisioning. It is a claim about which job they were tested for, and which job they failed in those tests.

What “exploration” means here

A decision system that only repeats yesterday’s winner will miss a better option that has not been tried enough. A system that always tries everything never settles. Exploration is the deliberate choice to gather information now so later choices get better.

Multi-armed bandits isolate that tradeoff. You choose among a small set of actions, see a reward for the chosen action only, and try to maximize total reward over time. Classic algorithms such as Thompson sampling keep a probability distribution over which option is best and sample from it. The randomness is not decoration. It is how the system keeps testing uncertain options without a hand-tuned exploration rate.

If you hand that same job to a language model and put the environment description and history in the prompt, you are testing in-context exploration. That is the setting studied by Krishnamurthy, Harris, Foster, Zhang, and Slivkins (2024).

What the in-context bandit tests actually showed

Krishnamurthy and coauthors put GPT-3.5, GPT-4, and Llama 2 into simple Bernoulli bandits. The problem description and interaction history lived entirely in the prompt. They tried many prompt designs: button-pressing versus ad-recommendation framing, raw versus summarized history, chain-of-thought, temperature, and hints to explore.

Their headline result is careful. Across their experiments, only one configuration produced satisfactory exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally summarized history presented as per-arm sufficient statistics. All other configurations they report did not robustly explore, including chain-of-thought setups that still used unsummarized history.

Two failure modes show up again and again in their diagnostics:

  1. Lock-in after early rounds. They call these suffix failures: after some early plays, the model never selects the best arm again. On a basic GPT-4 prompt, they report suffix failures in more than 60% of replicates in their longer-horizon illustration. That pattern looks more like greedy lock-in than like Thompson sampling or UCB.
  2. Uniform dithering. In other configurations the model keeps selecting all arms near-equally often and never narrows to the better ones. That is the opposite problem: motion without learning.

Both failures matter for a product. Lock-in stops learning about the winner. Uniform dithering wastes traffic forever. Neither is a reliable way to balance exploration and exploitation.

The authors are explicit about limits. These are synthetic bandits with a small number of arms and known gaps. Their “hard” instance uses five arms and a gap of 0.2; they also report an easier four-arm case. The successful run still needed outside structure: a suggestive prompt, an external summary of plays and average rewards, and reinforced chain-of-thought. Remove the external summary and the same stack fails in their tests. They note that external summarization may not even be available in richer settings such as contextual bandits with many contexts, where averaging per arm is no longer a cheap sufficient statistic.

They also stress sensitivity: small prompt changes move results, and the modifications interact, so it is hard to isolate one magic phrase. GPT-4 did better than GPT-3.5 in their tables; Llama 2 did worse, often with suffix-failure rates at or above greedy. That is useful context and still not a forecast for every future model or every marketing workflow.

So the fair reading is not “language models cannot decide.” It is: in these native, in-context tests, reliable exploration needed strong outside structure, and most prompt designs still failed.

Generated text is not a statistical policy

It helps to separate three objects that product pitches often mash together:

  • Content. Subject lines, offer copy, journey copy. Language models are already used here.
  • Setup. Naming a decision, listing valid actions, stating the reward event and window, describing eligibility and cooldown. A future assistant can draft that declaration from a conversation.
  • Live choice. For this person, right now, pick one approved option under a policy that still explores and that writes a decision log you can evaluate later.

BanditBuzz owns the third job with fixed statistical policies: Beta-Bernoulli Thompson sampling on the free tier, and a hierarchical linear Thompson sampler when per-person context is on. It does not generate creative. It returns or delivers a choice among the strings you approved. Your tools still build and send the experience. The landing page states the same boundary in plain terms: BanditBuzz does not invent or approve offers and creative; your marketing team sets those boundaries, and your customer tools still render and deliver.

That split matches the product surface. Options are opaque strings to the policy. Metadata may be stored for analysis. The learner does not read tone, brand voice, or prompt text. If you want a language model in the loop, put it upstream of the option list or downstream of the pick, not inside the sampler. A new option string that appears in a later decide call enters exploration under the same statistical rule; the model is not asked to invent a sampling schedule for it.

A 2025 survey by Bouneffouf and Féraud maps many proposed hybrids: bandits that help tune prompts or fine-tuning, language models that enrich context for bandits, routing, and feedback translation. Treat that paper as a source map, not as proof that any one product design works. The primary evidence for native in-context exploration remains the Krishnamurthy tests above.

Why a randomized log still matters

A policy that only emits a chosen action is hard to audit. A policy that also stores the probability of the chosen action under the logging rule creates a randomized dataset. That log supports offline checks of later candidate policies, including the doubly robust style of evaluation described by Dudík, Langford, and Li (2011).

Language-model lock-in and uniform dithering both damage that log. Lock-in under-samples the true winner. Uniform dithering never concentrates. Either way you get a history that is hard to trust for “what would a better policy have done?”

BanditBuzz therefore keeps the live choice inside a fixed algorithm, stores selection probabilities, and keeps a stable person-level holdout so the scoreboard can compare the whole live policy with a control. Those pieces are boring on purpose. They are the scaffolding that made the one successful Krishnamurthy configuration work, written as code instead of as a prompt trick.

Useful roles for a language model

Given the evidence, here is a responsibility split that stays honest.

Safe for a language model today

  • Draft option names and human-readable labels from a marketer’s description of the campaign.
  • Draft the first decision declaration: reward event, window, options including “nothing,” eligibility predicates, cooldown, and allowed send windows.
  • Explain scoreboard language in plain English after the numbers exist.
  • Help a human review whether two option strings are duplicates or whether a reward definition is too vague.
  • Generate creative variants that your team still reviews and then registers as ordinary options for the bandit to test.

Not safe as the sole live explorer

  • Asking the model, with raw history in the prompt, to pick the next arm without an external policy.
  • Treating chain-of-thought as a substitute for logged probabilities and a holdout.
  • Assuming that because the model can describe Thompson sampling, it will behave like Thompson sampling under stochastic rewards.

Still open research

  • Fine-tuning or tool use that forces correct sampling from a posterior.
  • External summaries or statistics that scale beyond tiny discrete arm sets.
  • Hybrid agents where the model proposes structure and a bandit still draws the action.

BanditBuzz’s current product stays on the conservative side of that list. A conversational setup assistant is named as future work in the product notes. Even then, the intended job is drafting configuration, not choosing live actions.

Model routing is a different, valid bandit problem

There is one adjacent place where language models and bandits fit cleanly together: choosing which model should answer a query.

Poon, Dai, Liu, Kong, Lui, and Zuo (2025) study online multi-LLM selection when the prompt evolves through a black-box process you cannot simulate or learn. Their answer is not “let one model decide everything.” It is a contextual bandit that makes myopic choices: pick a model for the current context, observe feedback, and avoid modeling the unlearnable future prompt path. They formalize myopic regret and give a LinUCB-style algorithm with budget and early-satisfaction variants.

That paper matters for two reasons. First, it shows a productive role for bandits around language models: routing, cost, and accuracy under feedback. Second, it supports a product design choice BanditBuzz already makes for lifecycle actions. When downstream funnel dynamics are opaque, optimize each declared decision against its own reward rather than inventing a full Markov model of the customer journey. Myopic contextual decisions are a feature when the next state cannot be modeled.

Routing which model writes a reply is not the same job as deciding whether this person should get offer_10_off, postcard, or nothing. Keep those problems separate.

How BanditBuzz draws the line

Put the pieces together as three roles:

  1. Language model (optional, outside the live loop). Helps humans name actions, draft setup, and explain results. May generate creative that humans approve.
  2. Bandit policy (BanditBuzz). Chooses among approved options with Thompson sampling, updates on matured rewards, and keeps exploration in the algorithm rather than in free-form generation.
  3. Execution system (your stack). Renders content, applies consent, sends or shows the experience, and reports delivery.

That arrangement matches the evidence we can cite without overclaiming. The in-context tests say native exploration is fragile without outside structure. The routing paper says bandits remain useful when language models are the arms. The survey says many hybrids are being tried; it does not replace primary tests.

If you are tempted to let a language model pick the action directly, ask for the same artifacts you would demand from any policy vendor: a stated rule for balancing exploration and exploitation, stored action probabilities, a holdout or other identification strategy, and a clear account of what happens when a new option appears. If the answer is “the model will figure it out from the transcript,” the Krishnamurthy results are a warning in the simplest setting we have.

None of this says language models have no place next to a decisioning API. It says the place should match the tested evidence: help with language and setup, leave live exploration to a policy you can name, log, and measure.

Use the language model where language is the work. Keep a policy where exploration is the work.