Exponential family

A single-parameter exponential family is a set of probability distributions whose probability density function can be expressed in the form:

$$f_X(x;\theta) = h(x) \text{ exp} \left [  \eta (\theta) T(x) - A(\theta)  \right ]$$

or equivalently

$$f_X(x;\theta) = h(x)g(\theta) \text{ exp} \left [  \eta (\theta) T(x)  \right ]$$

$$f_X(x;\theta) = \text{ exp} \left [  \eta (\theta) T(x) - A(\theta) + B(x)  \right ]$$

The definition in terms of one real-number parameter can be extended to one real-vector parameter $\mathbf{\theta} = (\theta_1, \theta_2, \dots, \theta_d)^T$. A family of distributions is said to belong to a vector exponential family if the probability density function can be written as

$$f_X(x;\mathbf{\theta}) = h(x) \text{ exp} \left [ \sum_{i=1}^s \eta_i(\mathbf{\theta})T_i(x) - A(\mathbf{\theta}) \right ]$$

Interpretation

• Sufficient statistic $T(x)$
  Heuristic definition: We say $T$ is a sufficient statistic if the statistician who knows the value of $T$ can do just as good a job of estimating the unknown parameter $\theta$ as the statistician who knows the entire random sample.
  Mathematical definition: A statistic $T=r(X_1, X_2, \dots, X_n)$ is a sufficient statistic if for each $t$, the conditional distribution of $X_1, X_2, \dots, X_n$ given $T=t$ and $\theta$ does not depend on $\theta$.
• Natural parameter $\eta$
  
• Logarithm of the normalization factor $A(\eta)$
  The normalization factor is a constant by which an everywhere non-negative function must be multiplied so the area under its graph is 1, e.g., to make it a probability density function or a probability mass function.
  (http://en.wikipedia.org/wiki/Normalizing_constant)

Examples

• Normal
• Exponential
• Log-normal
• Gamma
• Chi-squared
• Beta
• Dirichlet
• Bernoulli
• Categorical
• Poisson
• Geometric
• Inverse Gaussian
• Von Mises
• Von Mises-Fisher

References

• http://en.wikipedia.org/wiki/Exponential_family
• http://ko.wikipedia.org/wiki/%EC%A7%80%EC%88%98%EC%A1%B1
• http://sens.tistory.com/423
• Sufficient statistic: http://blog.naver.com/rupy400/130108738040
• Sufficient statistic: sufficient_statistic.pdf
• Sufficient statistic: http://blog.naver.com/weather12/130156410915
• http://blog.naver.com/pcapcoms/60100791155