Loss function is a method of evaluating how well your algorithm is modeling your dataset.
Why loss function is important?
You can't improve what you can't measure.
It is important for measuring the error of your model.
Types of loss functions?
Regression
Mse
Mae
Huber loss
Classification
Binary cross-entropy
Categorical cross-entropy
Hinge Loss
Autoencoders
- Kl divergence
GAN
Discriminator loss
Minmax gan loss
Object Detection
- Focal loss
Embedding
- Triplet loss
Loss function vs Cost Function?
Loss function is calculated on a single training example.
Cost function is calculated on the entire training dataset.
Regression:
Mean Square Error (MSE):
$$L = (y_i - \hat{y_i})^2$$
$$C = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y_i})^2$$
Mean Absolute Error(MAE):
$$L = |y_i - \hat{y_i}|$$
$$ C = \frac{1}{n}\sum_{i=1}^{n} |y_i - \hat{y_i}|$$
Huber Loss:
$$L_{\delta}= \left\{\begin{matrix} \frac{1}{2}(y - \hat{y})^{2} & if \left | (y - \hat{y}) \right | < \delta\\ \delta ((y - \hat{y}) - \frac1 2 \delta) & otherwise \end{matrix}\right.$$
Classification:
Binary Cross Entropy:
$$L = -{(y\log(\hat{y_i}) - (1 - y)\log(1 - \hat{y_i}))}$$
$$ C = -\frac{1}{n}\sum{(y\log(\hat{y_i}) + (1 - y)\log(1 - \hat{y_i}))}$$
Categorical Cross Entropy:
$$L = - \sum_{j=1}^k y_ilog(\hat{y_i})$$
where k is the number of classes in the data.
$$C = -\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{k} y_{ij}log(\hat{y_{ij}})$$
Note:
If you are working with regression use mse.
If the regression data has outliers use mae.
If you are working with binary classification data use bce.
If you are working with multi classification data use cce if you have few categories.
If you are working with multi classification data use spars categorical cross entropy if you have more categories.