Midterm Question: Why optimize for the g class, need to update other class’s corresponding parameters as well

Linear Classifier

The linear classifier is a hyperplane separates the space to $\pm 1$ .

But it has a limit: In cases where the data in nonlinear the linear classifier doesn’t work. Imagine the XOR problem.

Multi-Layer Neural Networks

The “deep” approach: stack multiple layers of linear transformations interspersed with nonlinearities.

Before:

f (x) = W_{1} x + b_{1}

Where $W_{1} \in R^{# of classes \times input dim}$ and $b_{1} \in R^{# of classes \times 1}$

Now:

f (x) = W_{2} g (W_{1} x + b_{1}) + b_{2}