5.1. Bayesian Inference & Priors

At the heart of Bayesian inference lies Bayes’ theorem:

where:

• $P(\theta)$ = Prior (what we believe before seeing data)
• $P(D | \theta)$ = Likelihood (how likely the data is, given our assumption)
• $P(\theta | D)$ = Posterior (updated belief after seeing data)
• $P(D)$ = Evidence (a normalizing constant ensuring probabilities sum to 1)

The posterior represents a compromise between what we thought and what we saw.

The Bayesian approach treats everything as uncertain — even parameters! Instead of fixed values, parameters are random variables whose distributions evolve with evidence.

Frequentists say: “The parameter is fixed, data is random.” Bayesians say: “The data is fixed, but the parameter is uncertain.”

This shift allows us to reason probabilistically about our uncertainty in models themselves — a powerful perspective in data science and ML.

Bayesian inference underpins many modern algorithms:

• Naïve Bayes: Uses priors over classes.
• Bayesian Networks: Graphical models built on conditional probabilities.
• Regularization: Can be seen as imposing a prior belief (e.g., “weights should be small”).
• Bayesian Optimization: Updates belief about a function’s shape using observed samples.

In ML terms, Bayesian reasoning = “belief updating with data.”

A conjugate prior is a prior distribution that, when combined with a likelihood, yields a posterior of the same family. This makes Bayesian updating analytically simple and elegant.

Example 1: Beta-Binomial Model (Coin Flips) If data comes from a Binomial likelihood (e.g., number of heads in coin flips), and the prior on the probability $p$ is Beta$(\alpha, \beta)$, then the posterior is also Beta:

where:

• $k$ = number of successes (heads)
• $n$ = number of trials

Interpretation: You start with “pseudo-counts” of $\alpha - 1$ heads and $\beta - 1$ tails. Each new observation updates these counts.

If $\alpha = \beta = 1$, that’s a uniform prior — no initial bias.

Example 2: Normal-Normal Model (Mean Estimation) For data with Gaussian noise, if prior on mean $\mu$ is $N(\mu_0, \sigma_0^2)$ and likelihood $Y_i \sim N(\mu, \sigma^2)$, then posterior is also normal:

After updating your posterior belief about $\theta$, you might ask:

“Given my new belief, what do I expect to see next?”

This is the posterior predictive distribution:

It averages predictions across all possible parameter values weighted by how plausible they are (from the posterior).

Interpretation: It’s a probabilistic “forecast” — accounting for both model uncertainty and data uncertainty.

Example: In the coin-flip Beta-Binomial case:

So your “updated” belief about the next event directly reflects your new counts of heads and tails.

Let’s revisit the coin-flip example:

1. Start with prior: $p \sim \text{Beta}(2, 2)$ (you think the coin is roughly fair).

2. Observe data: 3 heads, 1 tail → $k = 3, n = 4$.

3. Update posterior:

   $$ p | D \sim \text{Beta}(2 + 3, 2 + 1) = \text{Beta}(5, 3) $$

4. Posterior mean:

   $$ E[p|D] = \frac{5}{5 + 3} = 0.625 $$

So after 3 heads and 1 tail, your belief shifts — you now think the coin lands heads ~62.5% of the time.

If you had started with a stronger prior (say Beta(20,20)), the same data would move your belief much less.

• “The prior is just guessing.” → No — it encodes prior evidence, expert knowledge, or symmetry.
• “Bayesian = subjective.” → Priors can be subjective, but the updating process is objective.
• “Frequentist vs Bayesian is a competition.” → They’re complementary lenses — Bayesians quantify uncertainty explicitly, Frequentists rely on long-run frequency logic.