Activation functions are essential in neural networks because they introduce non-linearity, enabling the network to model complex relationships in the data. They help the network learn and capture non-linear patterns, making it more powerful and capable of solving a wide range of tasks, including classification, regression, and more.
Why activation functions are needed?
Activation functions are essential in neural networks because they introduce non-linearity, enabling the network to model complex relationships in the data. They help the network learn and capture non-linear patterns, making it more powerful and capable of solving a wide range of tasks, including classification, regression, and more.
All activation functions:
Sigmoid activation function:
$$\sigma(x) = \frac{1}{1 + e^{-x}}$$
Advantages
Output is in between [0,1].
Non-linear function.
Differentiable
Disadvantages
Saturating function
Non zero centered
Computationally expensive
Tanh activation function:
$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$
Advantages
Non-linear
Differentiable
Zero centered
Disadvantages
Saturating function (Vanishing gradient problem)
Computationally expensive
Relu activation function:
$$\text{ReLU}(x) = \max(0, x)$$
Advantages
Non-linear
Not Saturated in the positive region
Computationally inexpensive
Convergence is faster than Tanh and Sigmoid
Disadvantages
Not completely differentiable
None zero centered