What are multi-armed and contextual bandits?

A multi-armed bandit is a way to choose among options while learning which one works, shifting traffic toward the better performers as evidence arrives rather than waiting for a fixed test to finish. A contextual bandit does the same, but it also reads features of the person and the situation, so it can choose a different option for a different context. Both sit between a plain experiment and full personalization.

The name comes from a row of slot machines, the old “one-armed bandits.” Picture several machines with unknown payouts: you want to find the best one, but every pull you spend learning about a weak machine is a pull you did not spend on the best. That tension is the whole problem a bandit solves.

Exploration and exploitation

A bandit balances two things on every decision. Exploration means trying an option to gather evidence about how well it performs. Exploitation means using the option that currently looks best. Explore too much and you keep spending traffic on losers; exploit too early and you lock in a choice before you really know it is the best. A bandit algorithm, such as Thompson sampling or an upper-confidence-bound method, is just a rule for trading these off as data accumulates.

The practical payoff is that traffic moves toward winners on its own. Where an A/B test holds a losing variant at a fixed share until the test ends, a bandit tapers it off as the evidence mounts, which cuts the cost of showing options that do not work.

What the “contextual” part adds

A plain multi-armed bandit learns a single best option for everyone. A contextual bandit reads features of the user and the moment, such as the device, the time of day, the referring page, or recent behavior, and learns which option is best for which context. It can discover that one headline works in the morning and another at night, or that one offer suits returning visitors while another suits first-time ones.

That is what makes a contextual bandit a form of in-session personalization: it chooses per request, from the context in front of it, rather than applying one global winner. The canonical reference is the 2010 LinUCB work on personalized news recommendation, which framed article selection as a contextual-bandit problem.

Where bandits are used

Bandits fit decisions with many interchangeable options and a fast, clear signal: headline and creative selection, on-site offers and banners, and send-time optimization for messaging. They are most useful when the set of options changes often enough that running a fresh, full A/B test each time would be slow or wasteful.

They fit less well where the reward is slow or sparse, or where you need a clean, defensible estimate of the effect, because a bandit’s shifting assignment makes that estimate harder to produce.

Bandits and measurement

The cost of a bandit’s efficiency is statistical clarity. Because it changes how it assigns traffic as it learns, you cannot read its effect the way you read a randomized test with fixed groups. Estimating what a given policy would have earned, from data the bandit already logged, is its own discipline called off-policy evaluation. If a team runs bandits, ask how they value a policy and whether they keep any randomized holdout at all; without one, the headline numbers are easy to overstate. The measurement pillar covers this in full.

Bandits, experiments, and personalization

It helps to place a bandit against its neighbors. An A/B test finds the best single variant for everyone and then stops. Personalization chooses the best variant for each person or context. A contextual bandit is the thing in between: it learns while it serves, and when it reads context it is personalizing. The three get confused because all of them choose among variants, which is why they have their own comparison in the Comparisons cluster.