Source: Deep Learning Enthusiast
Editor: Deep Learning Natural Language Processing
Link: https://blog.csdn.net/shanglianlm/article/details/85019768
This article is about 1500 words, recommended reading time is 5 minutes
tensorflow and pytorch are very similar, here we take pytorch as an example.
19 Types of Loss Functions
1. L1 Loss L1Loss
Calculates the absolute difference between output and target.
torch.nn.L1Loss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
2. Mean Squared Error Loss MSELossCalculates the mean squared difference between output and target.
torch.nn.MSELoss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
3. Cross Entropy Loss CrossEntropyLossEffective for training classification problems with C categories. The optional parameter weight must be a 1D Tensor, weights will be assigned to each category. Very effective for unbalanced training sets.In multi-class tasks, softmax activation function + cross-entropy loss function is often used, as cross-entropy describes the difference between two probability distributions, while the neural network outputs a vector, not in the form of a probability distribution. Therefore, a softmax activation function is needed to normalize a vector into a probability distribution, and then the cross-entropy loss function is used to calculate the loss.
torch.nn.CrossEntropyLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.ignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
4. KL Divergence Loss KLDivLossCalculates the KL divergence between input and target. KL divergence can be used to measure the distance between different continuous distributions, and is very effective for direct regression in the space of continuous output distributions (discrete sampling).
torch.nn.KLDivLoss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
5. Binary Cross Entropy Loss BCELossCross-entropy calculation function for binary classification tasks. Used to measure reconstruction error, e.g., in autoencoders. Note that the target values t[i] must be in the range of 0 to 1.
torch.nn.BCELoss(weight=None, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length ‘nbatch’.
6. BCEWithLogitsLossThe BCEWithLogitsLoss function integrates the Sigmoid layer into the BCELoss class. This version is numerically more stable than using a simple Sigmoid layer and BCELoss separately, as combining these two operations into one layer can utilize the log-sum-exp trick for numerical stability.
torch.nn.BCEWithLogitsLoss(weight=None, reduction='mean', pos_weight=None)Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length ‘nbatch’.
7. Margin Ranking Loss
torch.nn.MarginRankingLoss(margin=0.0, reduction='mean')The loss function for each instance in the mini-batch is as follows:
Parameters:
margin: default value 0
8. Hinge Embedding Loss
torch.nn.HingeEmbeddingLoss(margin=1.0, reduction='mean')The loss function for each instance in the mini-batch is as follows:
Parameters:
margin: default value 1
9. Multi-label Classification Loss MultiLabelMarginLoss
torch.nn.MultiLabelMarginLoss(reduction='mean')The loss for each sample in the mini-batch is calculated as follows:
10. Smooth L1 Loss SmoothL1Loss
Also known as Huber loss function.
torch.nn.SmoothL1Loss(reduction='mean')
Where
11. Logistic Loss for Binary Classification SoftMarginLoss
torch.nn.SoftMarginLoss(reduction='mean')
12. Multi-label One-Versus-All Loss MultiLabelSoftMarginLoss
torch.nn.MultiLabelSoftMarginLoss(weight=None, reduction='mean')
13. Cosine Loss CosineEmbeddingLoss
torch.nn.CosineEmbeddingLoss(margin=0.0, reduction='mean')
Parameters:
margin: default value 0
14. Multi-Class Hinge Loss MultiMarginLoss
torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, reduction='mean')
Parameters:
p=1 or 2 Default: 1 margin: default value 1
15. Triplet Loss TripletMarginLoss
Similar to Siamese networks, specific example: give an A, then give B, C, and see which one, B or C, is more similar to A.
torch.nn.TripletMarginLoss(margin=1.0, p=2.0, eps=1e-06, swap=False, reduction='mean')
Where:
16. Connectionist Temporal Classification Loss CTCLossCTC loss can automatically align data that is not aligned, mainly used in training serialized data without prior alignment. For example, speech recognition, OCR recognition, etc.
torch.nn.CTCLoss(blank=0, reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
17. Negative Log Likelihood Loss NLLLossNegative log likelihood loss. Used for training classification problems with C categories.
torch.nn.NLLLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.ignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.
18. NLLLoss2dNegative log likelihood loss for image input. It calculates the negative log likelihood loss for each pixel.
torch.nn.NLLLoss2d(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.reduction – three values, none: no reduction; mean: returns the average of the loss; sum: returns the sum of the loss. Default: mean.
19. Poisson NLLLossNegative log likelihood loss with target values following a Poisson distribution.
torch.nn.PoissonNLLLoss(log_input=True, full=False, eps=1e-08, reduction='mean')Parameters:
log_input (bool, optional) – If set to True, the loss will be calculated according to the formula exp(input) – target * input. If set to False, the loss will be calculated as input – target * log(input + eps).
full (bool, optional) – Whether to compute the full loss, i.e., including the Stirling approximation term target * log(target) – target + 0.5 * log(2 * pi * target).
eps (float, optional) – Default value: 1e-8.
Editor: Yu Tengkai
Proofreader: Gong Li
