The maximum likelihood estimator is one of the most used estimators in statistics. In this article, we introduce this estimator and study its properties.

In a typical inference task, we have some data that we wish to understand better. The statistical approach is to model the source of these data as a random variable whose outcomes are produced with joint-probability where is an unknown parameter.

## Definitions

A maximum likelihood estimator for is an estimator that maximizes the probability of producing the sample we observed.

- Definition: likelihood
- The likelihood is the probability seen as a function of :

- Definition: MLE
- When the likelihood admits a unique global maximum, the MLE is:

In practice, we often maximize the log-likelihood instead of the likelihood. Since is an increasing function, this yields an equivalent solution.

The log-likelihood is noted :

Remarks:

- the likelihood is not the probability of ;
- maximizing the probability of is called “maximum a posteriori estimation”.

## Estimator performance

As explained in our primer on estimators, we first want to know if the MLE is consistent.

### Consistency

Under some regularity conditions on the density , the MLE is a consistent estimator, for instance:

- when and is concave;
- when and is continuously differentiable;
- when is from a -parameter exponential family.

### Asymptotic performance

Assuming an i.i.d. sample and under sufficient regularity of the distribution , the MLE has excellent asymptotical properties:

- Theorem
- For i.i.d. samples with sufficient regularity and assuming consistency, the asymptotic distribution of the MLE is:

Where:

is the Fisher information.

So, for large sample sizes :

- it is approximately normally distributed;
- approximately unbiased;
- approximately achieves the Cramer-Rao lower bound.

…What else?

What are those regularity conditions?

- is an open subset of (so that it always make sense for an estimator to have symmetric distribution around ).
- The support of is independent of (so that we can interchange integration and differentiation).
- .
- and .
- .
- and such that and:

### Other properties

The MLE is equivariant, which is very convenient in practice.

- Proposition: Equivariance of the MLE
- MLEs are equivariant: let a bijection. If is the MLE of , then is the MLE of :