The Beta-Binomial Model: A Ground-Up Reference

Bernoulli Trials and the Likelihood

Imagine that you have a coin that lands on heads with the probability $\theta$. You flip it $n$ times and observe $k$ heads. The probability of seeing exactly $k$ heads given $\theta$ is the Binomial likelihood:

$$P(k \mid \theta, n) = \binom{n}{k} \theta^k (1 - \theta)^{n - k}$$

This is your data-generating process. If $\theta$ is known, you can compute how likely any outcome is. But in practice $\theta$ is unknown, and you want to infer it from data.

The maximum likelihood estimate is $\hat{\theta} = k / n$, which outputs a single number. However, the tradeoff here is that a point estimate like this doesn’t account for uncertainty. With Bayesian statistics, we keep the answer distribution based, allowing us to estimate the uncertainty involved with calculating the likeliness of an outcome.

The Beta Prior

Before seeing any data, you have beliefs about $\theta$. If those beliefs are weak, you can choose a uniform (all events equally likely) prior or a very weak prior. This ensures that ultimately our decision is mostly informed by the data rather than shifted by our chosen prior.

The Beta distribution is a natural choice because it lives on $[0, 1]$ and can encode many shapes with just two parameters:

$$\theta \sim \text{Beta}(\alpha, \beta)$$

The density is:

$$f(\theta; \alpha, \beta) = \frac{\theta^{\alpha - 1} (1 - \theta)^{\beta - 1}}{B(\alpha, \beta)}$$

where $B$ is the Beta function (the normalizing constant here).

Alpha and beta are pseudocounts, basically imaginary prior observations. For example, $\text{Beta}(2, 5)$ behaves as if you have seen 2 heads and 5 tails before collecting real data.

Prior shape	When to use
$\alpha = \beta = 1$	Uniform — no prior preference, all $\theta$ equally likely
$\alpha > \beta$	Belief that $\theta$ is high (coin biased toward heads)
$\alpha < \beta$	Belief that $\theta$ is low (coin biased toward tails)
$\alpha, \beta \gg 1$	Strong prior certainty
$\alpha + \beta$ small	Weak prior — few pseudo-observations, so the data dominates quickly

The mean of $\text{Beta}(\alpha, \beta)$ is $\alpha / (\alpha + \beta)$, and the variance shrinks as the total pseudocount grows (intuitively this should make sense).

Conjugacy and the Posterior

The Beta distribution is conjugate to the Binomial likelihood. Conjugacy means the posterior is the same family as the prior. This is extremely useful because it means that multiplying our binomial likelihood by the prior Beta distribution just gives us another Beta distribution.

The update rule is strikingly simple:

$$\text{Beta}(\alpha, \beta) + (k \text{ heads in } n \text{ trials}) \longrightarrow \text{Beta}(\alpha + k, \beta + n - k)$$

Add the observed heads to $\alpha$, add the observed tails ($n - k$) to $\beta$.

The posterior mean becomes:

$$\frac{\alpha + k}{\alpha + \beta + n}$$

which is a weighted average of the prior mean $\alpha / (\alpha + \beta)$ and the observed proportion $k / n$. The weight is controlled by the total pseudocount $\alpha + \beta$. A larger prior sample size means the prior pulls harder against the data (exerting more influence on our results).

Using the Visualizer

The plot above (also on the homepage) lets you explore this update in real time.

$\alpha$ and $\beta$ sliders (PRIOR curves). These set your prior Beta distribution. The dashed curve shows the prior density. Start with $\alpha = 2, \beta = 5$ — a weak prior favoring tails. Increase both together to sharpen the prior. Increase $\alpha$ alone to shift weight toward heads.

$k$ and $n$ sliders (POSTERIOR curves). These represent the data you observe. Set $k$ to the number of heads seen out of $n$ trials. The solid curve is the posterior — the prior updated by your data. When $k / n$ differs from the prior mean, the posterior shifts. When $n$ is small, the prior dominates. As $n$ grows, the posterior converges toward $k / n$.

The formula caption below the sliders shows the exact posterior parameters:

$$\text{Beta}(\alpha + k, \beta + n - k)$$

Push $n$ to 30 with $k = 15$ and a prior of $\text{Beta}(2, 5)$ — the posterior will sit between the prior mean ($2/7 \approx 0.29$) and the observed proportion ($0.50$). With $n = 3$ and $k = 2$, the prior pulls much harder.

The Extension

The Beta-Binomial is the simplest case of a pattern that repeats across Bayesian statistics:

1. Choose a prior that captures what you know (or do not know) before seeing data
2. Write down the likelihood of the data given the parameters
3. Combine them via Bayes’ rule to get the posterior

The same recipe scales to linear regression (conjugate Normal-Inverse-Gamma priors), classification (Dirichlet-Multinomial), and Gaussian processes. It ultimately is the basic foundation of Bayesian inference. The ways in which we use Bayesian statistics are fundamentally just this loop repeated over and over in more domain specific contexts (drug discovery, clinical trials, personalized content recommendations, etc.).