26: Likelihood

Notation

Notation

Recall,

• Lower-case $\{x_1,x_2,x_3,...,x_n\}$ is a set of observations
• Upper-case $\{X_1,X_2,X_3,...,X_n\}$ is a set of random variables (i.e. a data set)
• Treating $\{X_1,X_2,...,X_n\}$ as a set of $n$ i.i.d. (independent and identically distributed) random variables is a common assumption.
• With independence, $$P(X_1,X_2,...,X_n)=P(X_1)\cdot P(X_2)\cdot ...\cdot P(X_n)$$
• Each individual probability is computed (at least theoretically) with a PDF (probability density function) $$P(x_i)=f_X(x_i)$$

Inverse

Inverse

Suppose that we have a sample of data $\{x_1,x_2,x_3,...,x_n\}$. Now we want to model with a probability distribution, but we need to figure out the distribution’s parameters. Let us think about this in a Bayesian way:

$$P(\text{model}|\text{data})={\displaystyle \frac{P(\text{data}|\text{model})\cdot P(\text{model})}{P(\text{data})}}$$

• $P(\text{model}|\text{data})$ is the posterior probability that we want
• $P(\text{data}|\text{model})$ is a likelihood
• Since the prior probability $P(\text{data})$ is a constant …

… we say that the posterior probability is proportional to the likelihood.

Likelihood

Definition

Likelihood Function

Let the likelihood function, in terms of a parameter $\theta$, be the joint probability

$$L(\theta )=P(X_1=x_1,X_2=x_2,...,X_n=x_n)=f_X(x_1)\cdot f_X(x_2)\cdots f_X(x_n)$$

or

$$L(\theta ;{\left\{x_i\right\}}_{i=1}^n)={\displaystyle \prod _{i=1}^n{f}_X(x_i)}$$

Example

Suppose that we have data for how long a certain type and brand of light bulb operated (in the same working conditions), and that data in months was

$$6,18,29,44,48$$ Goal: characterize the top 5 percent of light bulbs.

• Build the likelihood function assuming an exponential distribution.
• Compute the likelihood that $\mu =25$.
• Compute the likelihood that $\mu =50$.

Log Likelihood

Logarithms

Logarithms

You know that logarithms make large numbers smaller. More precisely, $$\ln (x)<x,x>1$$

Example: $\ln (1234)\approx 7.1180$

Did you know that logarithms make small numbers larger (in size). More precisely, $$|\ln (x)|>x,0<x<1$$

Example: $|\ln (0.1234)|\approx 2.0923$

From pre-calculus, recall the properties of logarithms: $$\ln (AB)=\ln (A)+\ln (B),\ln \left({\displaystyle \frac{A}{B}}\right)=\ln A-\ln B,\ln (A^c)=c\ln A$$

Example

For modeling with the exponential distribution, we saw that the likelihood function was

$$L(\lambda ;\{x_i{\}}_{i=1}^n)={\displaystyle \prod _{i=1}^n{f}_X(x_i)={\lambda }^n{e}^{-\lambda \sum x_i}}$$

We take the natural logarithm to compute the log-likelihood function

$$\ell (\lambda ;\{x_i{\}}_{i=1}^n)=\ln L(\lambda ;\{x_i{\}}_{i=1}^n)=n\ln \lambda -\lambda {\displaystyle \sum _{i=1}^n{x}_i}$$

• Compute the log-likelihood that $\mu =25$.
• Compute the log-likelihood that $\mu =50$.

Visuals

Looking Ahead

• WHW9

• Exam 2, Mon., Apr. 10

  • more information in weekly announcement

tweet source