Author: mingo_敏
Editor: Deep Learning Natural Language Processing
Link:https://blog.csdn.net/shanglianlm/article/details/85019768
Many of the loss functions in TensorFlow and PyTorch are similar; here we take PyTorch as an example.
19 Types of Loss Functions
1. L1 Loss L1Loss
Calculates the absolute difference between output and target.
torch.nn.L1Loss(reduction='mean')Parameters:
reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
2. Mean Squared Error Loss MSELossCalculates the mean squared error between output and target.
torch.nn.MSELoss(reduction='mean')Parameters:
reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
3. Cross Entropy Loss CrossEntropyLossEffective for training classification problems with C categories. The optional parameter weight must be a 1D Tensor, and the weights will be assigned to each category. Very effective for unbalanced training sets.In multi-class tasks, it is common to use the softmax activation function + cross-entropy loss function, as the cross-entropy describes the difference between two probability distributions, while the neural network outputs a vector, not in the form of a probability distribution. Therefore, the softmax activation function is needed to “normalize” a vector into the form of a probability distribution, and then use the cross-entropy loss function to calculate the loss.
torch.nn.CrossEntropyLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.ignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
4. KL Divergence Loss KLDivLossCalculates the KL divergence between input and target. KL divergence can be used to measure the distance between different continuous distributions and is very effective when performing direct regression on continuous output distribution space (discrete sampling).
torch.nn.KLDivLoss(reduction='mean')Parameters:
reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
5. Binary Cross Entropy Loss BCELossCross-entropy calculation function for binary classification tasks. Used to measure reconstruction error, such as in autoencoders. Note that the target values t[i] must be in the range of 0 to 1.
torch.nn.BCELoss(weight=None, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”.
6. BCEWithLogitsLossThe BCEWithLogitsLoss function integrates the Sigmoid layer into the BCELoss class. This version is numerically more stable than using a simple Sigmoid layer and BCELoss, as combining these two operations into one layer allows for numerical stability using the log-sum-exp trick.
torch.nn.BCEWithLogitsLoss(weight=None, reduction='mean', pos_weight=None)Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”.
7. Margin Ranking Loss
torch.nn.MarginRankingLoss(margin=0.0, reduction='mean')The loss function for each instance in a mini-batch is as follows:
Parameters:
margin: default value 0
8. Hinge Embedding Loss
torch.nn.HingeEmbeddingLoss(margin=1.0, reduction='mean')The loss function for each instance in a mini-batch is as follows:
Parameters:
margin: default value 1
9. Multi-Label Classification Loss MultiLabelMarginLoss
torch.nn.MultiLabelMarginLoss(reduction='mean')The loss for each sample in a mini-batch is calculated as follows:
10. Smooth L1 Loss SmoothL1Loss
Also known as Huber loss function.
torch.nn.SmoothL1Loss(reduction='mean')
Where
11. Logistic Loss for 2-Class SoftMarginLoss
torch.nn.SoftMarginLoss(reduction='mean')
12. Multi-Label One-Versus-All Loss MultiLabelSoftMarginLoss
torch.nn.MultiLabelSoftMarginLoss(weight=None, reduction='mean')
13. Cosine Loss CosineEmbeddingLoss
torch.nn.CosineEmbeddingLoss(margin=0.0, reduction='mean')
Parameters:
margin: default value 0
14. Multi-Class Hinge Loss MultiMarginLoss
torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, reduction='mean')
Parameters:
p=1 or 2, default value: 1margin: default value 1
15. Triplet Loss TripletMarginLoss
Similar to Siamese networks, a specific example: give an A, then give B, C, and see who is more similar to A.
torch.nn.TripletMarginLoss(margin=1.0, p=2.0, eps=1e-06, swap=False, reduction='mean')
Where:
16. Connectionist Temporal Classification Loss CTCLossCTC Connectionist Temporal Classification Loss can automatically align data that is not aligned, mainly used for training serialized data without prior alignment, such as speech recognition, OCR recognition, etc.
torch.nn.CTCLoss(blank=0, reduction='mean')Parameters:
reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
17. Negative Log Likelihood Loss NLLLossNegative log likelihood loss. Used for training classification problems with C categories.
torch.nn.NLLLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.ignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.
18. NLLLoss2dNegative log likelihood loss for image input. It calculates the negative log likelihood loss for each pixel.
torch.nn.NLLLoss2d(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length C.reduction – three values: none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
19. Poisson NLL LossNegative log likelihood loss for target values following a Poisson distribution.
torch.nn.PoissonNLLLoss(log_input=True, full=False, eps=1e-08, reduction='mean')Parameters:log_input (bool, optional) – If set to True, the loss will be calculated using the formula exp(input) – target * input; if set to False, the loss will be calculated as input – target * log(input + eps).full (bool, optional) – Whether to calculate the full loss, i.e., including the Stirling approximation term target * log(target) – target + 0.5 * log(2 * pi * target).eps (float, optional) – Default value: 1e-8
