18: Linear Operators

Previously, in Math 32,

• $X$ is a random variable with PDF $f$
• $Y$ is a random variable with PDF $g$

$$\text{E}[X]={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\cdot f(x)dx},\text{E}[Y]={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}y\cdot g(y)dy}$$

$$\text{Var}(X)=\text{E}[X^2]-{(\text{E}[X])}^2,\text{Var}(Y)=\text{E}[Y^2]-{(\text{E}[Y])}^2$$

Let $a$, $b$, and $c$ be constants.

Linear Operators

Linear Operators

We say that $L$ is a linear operator if

$$\begin{array}{rcl}L(af(x))& =& aL(f(x))\\ L(f(x)+g(x))& =& L(f(x))+L(g(x))\end{array}$$

Linear Operators

Loosely translated, $L$ is a linear operator if

• we can factor out a scalar multiple
• we can split the operator across a sum or difference

Calculus Review

• Show that the derivative operator is a linear operator.
  
• Show that the integral operator is a linear operator.

Claim: the derivative operator is a linear operator

Proof

$$\begin{array}{rcl}{\displaystyle \frac{d}{dx}(af(x)+bg(x))}& =& {\displaystyle \frac{d}{dx}af(x)+{\displaystyle \frac{d}{dx}bg(x)}}\\ & =& a{\displaystyle \frac{d}{dx}f(x)+b{\displaystyle \frac{d}{dx}g(x)}}\end{array}$$

$${\displaystyle \frac{d}{dx}(af(x)+bg(x))=a{\displaystyle \frac{d}{dx}f(x)+b{\displaystyle \frac{d}{dx}g(x),}}}$$

so ${\displaystyle \frac{d}{dx}}$ is a linear operator.

Claim: the integral operator is a linear operator

Proof

$$\begin{array}{rcl}{\displaystyle \int (af(x)+bg(x))dx}& =& {\displaystyle \int af(x)dx+{\displaystyle \int bg(x)dx}}\\ & =& a{\displaystyle \int f(x)dx+b{\displaystyle \int g(x)dx}}\end{array}$$

$${\displaystyle \int (af(x)+bg(x))dx=a{\displaystyle \int f(x)dx+b{\displaystyle \int g(x)dx,}}}$$

so ${\displaystyle \int }$ is a linear operator.

Expected Value

Is the expectation operator $\text{E}$ a linear operator?

Claim: $\text{E}[aX]=a\text{E}[X]$

Proof

$$\text{E}[aX]={\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}ax\cdot f(x)dx=a{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\cdot f(x)dx}=a\text{E}[X]}$$

We have shown that we can factor out a scalar multiple across the expectation operator.

Claim: $\text{E}[X+c]=\text{E}[X]+c$

Proof

$$\begin{array}{rcl}\text{E}[X+c]& =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}(x+c)\cdot f(x)dx}\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\cdot f(x)dx}+{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}c\cdot f(x)dx}\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\cdot f(x)dx}+c{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)dx}\\ & =& \text{E}[X]+c\end{array}$$

We have shown that a horizontal shift of $c$ units in the data also affects the expected value by $c$ units

Claim: $\text{E}[X+Y]=\text{E}[X]+\text{E}[Y]$

Proof

$$\begin{array}{rcl}\text{E}[X+Y]& =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}(x+y)\cdot f(x,y)dydx\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}x\cdot f(x,y)dydx+{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}}y\cdot f(x,y)dxdy\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\cdot f_X(x)dx}+{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}y\cdot f_Y(y)dy}\\ & =& \text{E}[X]+\text{E}[Y]\end{array}$$

We have shown that the expected value of a sum \ is the sum of the expected values.

Conclusion

Combining the above results, since

$$\text{E}[aX+bY]=a\text{E}[X]+b\text{E}[Y]$$

we have shown that the expectation operator $\text{E}$ is a linear operator.

Also,

$$\text{E}[aX+bY+c]=a\text{E}[X]+b\text{E}[Y]+c$$

Variance

Is the variance $\text{Var}(X)$ function a linear operator?

Claim: $\text{Var}(aX)=a\text{Var}(X)$

Counterpoint:

Recall the ``practical formula for variance’’

$$\text{Var}(X)=\text{E}[X^2]-{(\text{E}[X])}^2$$

and tracking the scaling factor proceeds as follows

$$\begin{array}{rcl}\text{Var}(aX)& =& \text{E}[(aX{)}^2]-{(\text{E}[aX])}^2\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}(ax{)}^2\cdot f(x)dx-{(a\text{E}[X])}^2}\\ & =& {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{a}^2{x}^2\cdot f(x)dx-a^2{(\text{E}[X])}^2}\\ & =& a^2\left[{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{x}^2\cdot f(x)dx-{(\text{E}[X])}^2}\right]\\ & =& a^2(\text{E}[X^2]-{(\text{E}[X])}^2)\\ & =& a^2\text{Var}(X)\end{array}$$

When factoring out a scalar from the variance function, the factor is squared.

Furthermore, since $\text{Var}(aX)\ne a\text{Var}(X)$, we have shown that the variance function is not a linear operator.

Claim: $\text{Var}(X+c)=\text{Var}(X)+c$

Counterpoint:

Recall the ``practical formula for variance’’

$$\begin{array}{rcl}\text{Var}(X+c)& =& \text{E}[(X+c{)}^2]-{(\text{E}[X+c])}^2\\ & =& \text{E}[X^2+2cX+c^2]-{(\text{E}[X]+c)}^2\\ & =& \text{E}[X^2]+\text{E}[2cX]+\text{E}[c^2]-{(\text{E}[X])}^2-2c\text{E}[X]-c^2\\ & =& \text{E}[X^2]+2c\text{E}[X]+c^2-{(\text{E}[X])}^2-2c\text{E}[X]-c^2\\ & =& \text{E}[X^2]-{(\text{E}[X])}^2\\ & =& \text{Var}(X)\end{array}$$

We have shown that $\text{Var}(X+c)=\text{Var}(X)$. That is, variance is not affected by a horizontal shift (phase shift)!

Furthermore, since $\text{Var}(X+c)\ne \text{Var}(X)+c$, we have shown that the variance function is not a linear operator.

Claim: $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$

Counterpoint:

$$\begin{array}{rcl}\text{Var}(X+Y)& =& \text{E}[(X+Y{)}^2]-{(\text{E}[X+Y])}^2\\ & =& \text{E}[X^2+2XY+Y^2]-{(\text{E}[X]+\text{E}[Y])}^2\\ & =& \text{E}[X^2]+\text{E}[2XY]+\text{E}[Y^2]-{(\text{E}[X])}^2+2\text{E}[X]\text{E}[Y]+{(\text{E}[Y])}^2\\ & =& \text{E}[X^2]-{(\text{E}[X])}^2+\text{E}[Y^2]-{(\text{E}[Y])}^2+2\text{E}[XY]-2\text{E}[X]\text{E}[Y]\\ & =& \text{Var}(X)+\text{Var}(Y)+2\left(\text{E}[XY]-\text{E}[X]\text{E}[Y]\right)\end{array}$$

We have shown that $\text{Var}(X+Y)\ne \text{Var}(X)+\text{Var}(Y)$. That is, the variance of the sum is not the sum of the variances (unless …?)

Furthermore, since $\text{Var}(X+Y)\ne \text{Var}(X)+\text{Var}(Y)$, we have shown that the variance function is not a linear operator.

Conclusion

We have shown that the variance function is not a linear operator.

Next time: working with

$$\text{E}[XY]-\text{E}[X]\text{E}[Y]$$

which is called the covariance!

Looking Ahead

• due Fri., Mar. 10:

  • WHW7
  • LHW6
  • Internet Connection (survey)

• Exam 2 will be on Mon., Apr. 10

• no lecture on Mar. 10, Mar. 24

tweet source