Lecture 3

The Regression §

Suppose we observe a real valued input x and we wish to use the observation to predict the value of a real-valued target variable y

Applications:

• Financial domain: house price, stock market, etc
• Healthcare domain: vital measurements are in real value

House Buying Example Problem §

Problem: You want to predict how much a house is worth. Target: Actual price for house as denoted as y Input Features:

• $x_1:$ number of bedrooms
• $x_2:$ number of bathrooms
• $x_3:$ living sqft
• etc. etc.

Predictive Model §

• Input: $x=[x_1,x_2,...,x_d]\in {\mathbb{R}}^d$
• Output: $y\in R$
• Functional Form:
  • $f:\mathbb{R}\to \mathbb{R}$ infinitely many
  • $f(x)=w^{T}x+b,w\in \mathbb{R},b\in \mathbb{R}$

Note: Another form this function can take is $f(x)=Wx$.

To convert between $f(x)=w^{T}x+b$ to $f(x)=Wx$:

$$\begin{array}{r}{x}^{\prime }=\left[\begin{array}{c}x\\ 1\end{array}\right]\\ W=\left[\begin{array}{c}w\\ b\end{array}\right]\end{array}$$

How to Find a Good Model? §

Criteria: Compare the true value vs the estimated value via a squared loss

$$\begin{array}{rl}X& :\text{Input vector}\\ B& :\text{Bias}\end{array}$$

$$\underset{\theta }{\min }\sum _{i=1}^N||f(x_i;\theta )-y_i|{|}_2^2$$

Linear Regression:

$$\underset{w,b}{\min }\sum _{i=1}^N||w^T{x}_i+b-y_i|{|}_2^{2}w\in {\mathbb{R}}^d,b\in \mathbb{R}$$

Matrix Form:

$$\underset{w,b}{\min }||Xw+b-y|{|}_2^{2}w\in {\mathbb{R}}^d,X\in {\mathbb{R}}^{n\times d},y\in \mathbb{R}$$

Note that $b$ may be excluded, it can be counted as a part of $w$.

Re-Parametrization §

Tilde implies a re-parametrization, that the bias term is being included in the w and x term.

$$\begin{array}{rl}f(x)& =w^{T}x+b,w\in {\mathbb{R}}^d,b\in \mathbb{R}\\ f(\tilde{x})& ={\tilde{w}}^T\tilde{x},\tilde{w}\in {\mathbb{R}}^{d+1},\tilde{x}=[x+1]\in {\mathbb{R}}^{d+1}\end{array}$$

Matrix Derivative §

Gradient of a function §

In many machine learning problems, the objective involves a function
that takes a vector of variables as input, e.g., $f(w)=w^{T}x=w_1{x}_1+w_2{x}_2$

Suppose the first partial derivatives of 𝑓 exist, the gradient of 𝑓 (denoted
by ∇𝑓 ) is the vector field whose ith component is the ith partial derivative. (Jacobian)

Solving Least Squares Problem §

Given:

$$\underset{\tilde{w}}{\min }L(\tilde{w})=||\tilde{X}\tilde{w}-y|{|}_2^2,\tilde{w}\in {\mathbb{R}}^{d+1},\tilde{X}\in {\mathbb{R}}^{n\times d+1},y\in {\mathbb{R}}^n$$

Use these equations to solve for $\tilde{w}\ast$:

$$\begin{array}{rl}& 1.||a|{|}_2^2=a^{T}a,a\in {\mathbb{R}}^n\\ & 2.(AB{)}^T=B^T{A}^T\\ & 3.{\nabla }_w(w^{T}Aw)=(A+A^T)w\end{array}$$

Solution:

$$\begin{array}{rlr}L(\tilde{w})& =(\tilde{X}\tilde{w}-y{)}^T(\tilde{X}\tilde{w}-y)& (\text{Eq. 1})\\ & =(\tilde{X}\tilde{w}{)}^T\tilde{X}\tilde{w}-(\tilde{X}\tilde{w}{)}^{T}y-y^T\tilde{X}\tilde{w}+y^{T}y& (\text{distributed transpose})\\ & ={\tilde{w}}^T{\tilde{X}}^T\tilde{X}\tilde{w}-2y^T\tilde{X}\tilde{w}+y^{T}y& (\text{Eq. 2})\\ {\nabla }_{w}L& =({\tilde{X}}^T\tilde{x}+({\tilde{X}}^T\tilde{x}{)}^T)\tilde{w}-2(y^T\tilde{x}{)}^T& (\text{Eq. 3})\\ & =2{\tilde{X}}^T\tilde{X}\tilde{w}-2{\tilde{X}}^T\tilde{y}& (\text{Eq. 2})\end{array}$$

set the gradient equal to zero to find the minimum:

$$\begin{array}{rl}& \\ {\nabla }_{w}L& =2{\tilde{X}}^T\tilde{X}\tilde{w}-2{\tilde{X}}^T\tilde{y}=0\\ & {\tilde{X}}^T\tilde{X}\tilde{w}-{\tilde{X}}^T\tilde{y}=0\\ \tilde{w}& =({\tilde{X}}^T\tilde{X}{)}^{-1}({\tilde{X}}^T\tilde{y})\end{array}$$

Note: Rank deficiency problem. When ${\tilde{X}}^T\tilde{X}$ is close to singular, we would have numerical problems. The solution is to add regularization (Ridge Regression).

$$\underset{\tilde{w}}{\min }L(\tilde{w})=||\tilde{X}\tilde{w}-y|{|}_2^2+\lambda ||\tilde{w}|{|}_2^2$$

Ridge Regression §

Ridge Regression has the formula $\underset{\tilde{w}}{\min }L(\tilde{w})=||\tilde{X}\tilde{w}-y|{|}_2^2+\lambda ||\tilde{w}|{|}_2^2$ where $\lambda >0$

Solving for the analytical solution:

$$\begin{array}{rlr}\underset{\tilde{w}}{\min }L(\tilde{w})& =||\tilde{X}\tilde{w}-y|{|}_2^2+\lambda ||\tilde{w}|{|}_2^2& \\ & ={\tilde{w}}^T{\tilde{X}}^T\tilde{X}\tilde{w}-2y^T\tilde{X}\tilde{w}+y^{T}y+\lambda ||\tilde{w}|{|}_2^2& (\text{From linear regression})\\ & ={\tilde{w}}^T{\tilde{X}}^T\tilde{X}\tilde{w}-2y^T\tilde{X}\tilde{w}+y^{T}y+\lambda {\tilde{w}}^T\tilde{w}& (\text{Eq. 1})\\ {\nabla }_{\tilde{w}}L(\tilde{w})& =({\tilde{X}}^T\tilde{X}+({\tilde{X}}^T\tilde{X}{)}^T)\tilde{w}-2y^T\tilde{X}+2\lambda \tilde{w}& \\ & =2{\tilde{X}}^T\tilde{X}\tilde{w}-2y^T\tilde{X}+2\lambda \tilde{w}& \end{array}$$

set the gradient equal to zero to find the minimum:

$$\begin{array}{rl}0& =2{\tilde{X}}^T\tilde{X}\tilde{w}-2y^T\tilde{X}+2\lambda \tilde{w}\\ y^T\tilde{X}& ={\tilde{X}}^T\tilde{X}\tilde{w}+\lambda \tilde{w}\\ y^T\tilde{X}& =({\tilde{X}}^T\tilde{X}+\lambda I)\tilde{w}\\ \tilde{w}& =y^T\tilde{X}({\tilde{X}}^T\tilde{X}+\lambda I{)}^{-1}\end{array}$$

The gradient also has the formula ${\nabla }_{\tilde{w}}L(\tilde{w})=\frac{-2}{N}{X}^T(y-X\tilde{w})+2\lambda \tilde{w}$