Conjugate prior

Poll without page-refresh

	del.icio.us
	Digg
	Furl
	Reddit
	Rojo
	Add to OnlyWire

In Bayesian probability theory, a class of prior probability distributions p(θ) is said to be conjugate to a class of likelihood functions p(x|θ) if the resulting posterior distributions p(θ|x) are in the same family as p(θ). For example, the Gaussian family is conjugate to itself (or self-conjugate): if the likelihood function is Gaussian, choosing a Gaussian prior will ensure that the posterior distribution is also Gaussian. The concept, as well as the term "conjugate prior", were introduced by Howard Raiffa and Robert Schlaifer in their work on Bayesian decision theory.^[1] A similar concept had been discovered independently by George Alfred Barnard.^[2]

Consider the general problem of inferring a distribution for a parameter θ given some datum or data x. From Bayes' theorem, the posterior distribution is calculated from the prior p(θ) and the likelihood function $\theta \mapsto p(x\mid\theta)\!$ as

$p(\theta|x) = \frac{p(x|\theta) \, p(\theta)} {\int p(x|\theta) \, p(\theta) \, d\theta}. \!$

Let the likelihood function be considered fixed; the likelihood function is usually well-determined from a statement of the data-generating process. It is clear that different choices of the prior distribution p(θ) may make the integral more or less difficult to calculate, and the product p(x|θ) × p(θ) may take one algebraic form or another. For certain choices of the prior, the posterior has the same algebraic form as the prior (generally with different parameters). Such a choice is a conjugate prior.

A conjugate prior is an algebraic convenience: otherwise a difficult numerical integration may be necessary.

All members of the exponential family have conjugate priors. See Gelman et al.^[3] for a catalog.

1 Example
2 Table of conjugate distributions
- 2.1 Discrete likelihood distributions
- 2.2 Continuous likelihood distributions
3 References
4 External links

Example

For a random variable which is a Bernoulli trial with unknown probability of success q in [0,1], the usual conjugate prior is the beta distribution with

$p(q=x) = {x^{\alpha-1}(1-x)^{\beta-1} \over \Beta(\alpha,\beta)}$

where $α$ and $β$ are chosen to reflect any existing belief or information ( $α$ = 1 and $β$ = 1 would give a uniform distribution) and Β( $α$ , $β$ ) is the Beta function acting as a normalising constant.

If we then sample this random variable and get s successes and f failures, we have

$P(s,f|q=x) = {s+f \choose s} x^s(1-x)^f,$

$p(q=x|s,f) = {{{s+f \choose s} x^{s+\alpha-1}(1-x)^{f+\beta-1} / \Beta(\alpha,\beta)} \over \int_{y=0}^1 \left({s+f \choose s} y^{s+\alpha-1}(1-y)^{f+\beta-1} / \Beta(\alpha,\beta)\right) dy} = {x^{s+\alpha-1}(1-x)^{f+\beta-1} \over \Beta(s+\alpha,f+\beta)} ,$

which is another Beta distribution with a simple change to the parameters. This posterior distribution could then be used as the prior for more samples, with the parameters simply adding each extra piece of information as it comes.

Table of conjugate distributions

Let n denote the number of observations

Discrete likelihood distributions

Likelihood	Model parameters	Conjugate prior distribution	Prior hyperparameters	Posterior hyperparameters
Bernoulli	p (probability)	Beta	$\alpha,\, \beta\!$	$\alpha + \sum_{i=1}^n x_i,\, \beta + n - \sum_{i=1}^n x_i\!$
Binomial	p (probability)	Beta	$\alpha,\, \beta\!$	$\alpha + \sum_{i=1}^n x_i,\, \beta + \sum_{i=1}^nN_i - x_i\!$
Negative Binomial	p (probability)	Beta	$\alpha,\, \beta\!$	$\alpha + rn,\, \beta + \sum_{i=1}^n x_i - rn\!$
Poisson	λ (rate)	Gamma	$\alpha,\, \beta\!$ ^[4]	$\alpha + \sum_{i=1}^n x_i,\ \beta + n\!$
Multinomial	p (probability vector)	Dirichlet	$\vec{\alpha}\!$	$\vec{\alpha}+\sum_{i=1}^n\vec{x}^{\,(i)}\!$
Geometric	p₀ (probability)	Beta	$\alpha,\, \beta\!$	$\alpha + n,\, \beta + \sum_{i=1}^n x_i\!$

Continuous likelihood distributions

Likelihood	Model parameters	Conjugate prior distribution	Prior hyperparameters	Posterior hyperparameters
Uniform	$U(0,\theta)\!$	Pareto	$x_{m},\, k\!$	$\max\{\,x_{(n)},x_{m}\},\, k+n\!$
Exponential	λ (rate)	Gamma	$\alpha,\, \beta\!$ ^[4]	$\alpha+n,\, \beta+\sum_{i=1}^n x_i\!$
Normal with known variance σ²	μ (mean)	Normal	$\mu_0,\, \sigma_0^2\!$	$(\frac{\mu_0}{\sigma_0^2} + \frac{\sum_{i=1}^n x_i}{\sigma^2})/(\frac{1}{\sigma_0^2} + \frac{n}{\sigma^2}),\, (\frac{1}{\sigma_0^2} + \frac{n}{\sigma^2})^{-1}$
Normal with known precision τ	μ (mean)	Normal	$\mu_0,\, \tau_0\!$	$(\tau_0 \mu_0 + \tau \sum_{i=1}^n x_i)/(\tau_0 + n \tau),\, \tau_0 + n \tau$
Normal with known mean μ	σ² (variance)	Scaled inverse chi-square	$\nu,\, \sigma_0^2\!$	$\nu+n,\, \frac{\nu\sigma_0^2 + \sum_{i=1}^n (x_i-\mu)^2}{\nu+n}\!$
Normal with known mean μ	τ (precision)	Gamma	$\alpha,\, \beta\!$ ^[4]	$\alpha + \frac{n}{2},\, \beta + \frac{\sum_{i=1}^n (x_i-\mu)^2}{2}\!$
Normal with known mean μ	σ² (variance)	Inverse Gamma Distribution	$\mathbf{\alpha,\, \beta}$ ^[5]	$\mathbf{\alpha}+\frac{n}{2},\, \mathbf{\beta} + \frac{\sum_{i=1}^n{(x_i-\mu)^2}}{2}$ , where $\bar{x}$ is sample mean
Normal	μ and σ² Assuming exchangeability	Normal-scaled inverse gamma	$\lambda ,\, \nu ,\, \alpha ,\, \beta$	$\frac{n\bar{x}+\nu\lambda}{n+\nu} ,\, \nu+n,\, \alpha+\frac{n}{2} ,\, \beta + \tfrac{1}{2} \sum_{i=1}^n (x_i - \bar{x})^2 + \frac{n\nu}{n+\nu}\frac{(\bar{x}-\lambda)^2}{2}$
Normal	μ and τ Assuming exchangeability	Normal-gamma	$\lambda,\, \gamma,\, \alpha,\, \beta\$	$\frac{\lambda \gamma+n\bar{x}}{\gamma+n},\, \gamma+n,\, \alpha+\frac{n}{2},\, \beta+\frac{nS^2}{2}+\frac{n\gamma(\bar{x}-\lambda)^2}{2(n+\gamma)}$ , where $\bar{x}$ is sample mean and $S 2$ is the sample variance.
Multivariate normal with known covariance matrix	μ (mean vector)	Multivariate normal	$\mathbf{\mu}_0,\, \Sigma_0$	$\left(\Sigma_0^{-1} + n\Sigma^{-1}\right)^{-1}\left( \Sigma_0^{-1}\mu_0 + n \Sigma^{-1} \bar{x} \right),\, \left(\Sigma_0^{-1} + n\Sigma^{-1}\right)^{-1}$ , where $\bar{x}$ is the sample mean.
Multivariate normal with known precision matrix	μ (mean vector)	Multivariate normal	$\mathbf{\mu}_0,\, \Lambda_0$	$\left(\Lambda_0 + n\Lambda\right)^{-1}\left( \Lambda_0\mu_0 + n \Lambda \bar{x} \right),\, \left(\Lambda_0 + n\Lambda\right)^{-1}$ , where $\bar{x}$ is the sample mean.
Multivariate normal with known mean	Σ (covariance matrix)	Inverse-Wishart	$\kappa ,\, \Psi$	$n+\kappa ,\, \Psi + \sum_{i=1}^n (x_i - \mu) (x_i - \mu)^T$
Multivariate normal with known mean	Λ (precision matrix)	Wishart
Multivariate normal	μ (mean vector) and Σ (covariance matrix)	Normal-Inverse-Wishart	$\lambda ,\, \nu ,\, \kappa ,\, \Psi$	$\frac{n\bar{x}+\nu\lambda}{n+\nu} ,\, n+\nu,\, n+\kappa ,\, \Psi + \sum_{i=1}^n (x_i - \bar{x}) (x_i - \bar{x})^T + \frac{n\nu}{n+\nu}(\bar{x}-\lambda)(\bar{x}-\lambda)^T$ , where $\bar{x}$ is the sample mean
Multivariate normal	μ (mean vector) and Λ (precision matrix)	Normal-Wishart	$\kappa_0,\, \mathbf{\mu}_0,\, \nu_0,\, \Lambda_0$	$(\kappa_0 + n),\, \frac{\kappa_0}{\kappa_0 + n} \mathbf{\mu}_0 + \frac{n}{\kappa_0 + n} \bar{x},\, (\nu_0 + n),\, \left( \Lambda_0^{-1} + C + \frac{\kappa_0 n}{\kappa_0 + n} (\bar{x} - \mathbf{\mu}_0) (\bar{x} - \mathbf{\mu}_0)^T \right)^{-1}$ where $\bar{x}$ is the sample mean and $C = \sum_{i=1}^n (x_i - \bar{x}) (x_i - \bar{x})^T$ .
Pareto	k (shape)	Gamma	$\alpha,\, \beta\!$	$\alpha+n,\, \beta+\sum_{i=1}^n \ln\frac{x_i}{x_{\mathrm{m}}}\!$
Pareto	x_m (location)	Pareto	$x_0,\, k_0\!$	$x_0,\, k_0-kn \!$ where $k_0 > kn\!$ .
Gamma with known shape α	β (inverse scale)	Gamma	$\alpha_0,\, \beta_0\!$	$\alpha_0+n\alpha,\, \beta_0+\sum_{i=1}^n x_i\!$
Inverse Gamma with known shape α	β (inverse scale)	Gamma	$\alpha_0,\, \beta_0\!$	$\alpha_0+n\alpha,\, \beta_0+\sum_{i=1}^n \frac{1}{x_i}\!$
Gamma ^[6]	α (shape), β (inverse scale)	$\frac{1}{Z} \frac{p^{\alpha-1} e^{-\beta q}}{\Gamma(\alpha)^r \beta^{-\alpha s}}$	$p,\, q,\, r,\, s \!$	$p \prod_{i=1}^n x_i,\, q + \sum_{i=1}^n x_i,\, r + n,\, s + n \!$

References

^ Howard Raiffa and Robert Schlaifer. Applied Statistical Decision Theory. Division of Research, Graduate School of Business Administration, Harvard University, 1961.
^ Jeff Miller et al. Earliest Known Uses of Some of the Words of Mathematics, "conjugate prior distributions". Electronic document, revision of November 13, 2005, retrieved December 2, 2005.
^ Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis, 2nd edition. CRC Press, 2003. ISBN 1-58488-388-X.
^ ^a ^b ^c β is rate or inverse scale. In parameterization of Gamma distribution,θ = 1/β and k = α.
^ In terms of the Inverse Gamma Distribution, $β$ is a Scale parameter
^ Fink, D. 1995 A Compendium of Conjugate Priors. In progress report: Extension and enhancement of methods for setting data quality objectives. (DOE contract 95‑831).