What is the use of activation function
Brief intro to Neural Network
The following picture illustrates the basic structure of neural network.
Extracted featrures are passed into the net as input.
# of input node = # of extracted feature types
Why we need activation
For a classifiying issue:
Without activation
If there is only one-layer(input-output), the output is
\(output = \sum input + loss\)
If there rae two-layers(with one hidden layer), the output is
\(output = \sum_ {hidden}(\sum_ {input} input+loss_1)+loss_2\)
So bascically, we can derive that no matter how many layers you add into the net, without activation, the net can only produce a linear classifier.
With activation
The aim of activation is to transfer a linear classifer to a non-linear classifer.
The basic actication we use is: \[ \begin{equation} {y}=f(x) =\left\{ \begin{array}{ll} 1 & x>0 \\ 0 & x<0 \end{array}\right. \end{equation} \] It is same as nerve cell--if the cell is activated it produce a signal, otherwise, it does nothing.
More then, what we use is sigmoid function. \[ \sigma(y) = \frac 1 {1+e^{-y}} \] (we prefer to ues \(y\), because actication is added inside the net which means the input of activation is the output of prior layer)
