Click on the top “MLNLP” to select “star” public account
Important content delivered at the first time
Author: mingo_敏
Editor: Deep Learning Natural Language Processing Editor zenRRan
Link:
https://blog.csdn.net/shanglianlm/article/details/85019768
TensorFlow and PyTorch have many similarities, here we take PyTorch as an example.
19 Loss Functions
1. L1 Loss L1Loss
Calculates the absolute difference between output and target.
torch.nn.L1Loss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
2 Mean Squared Error Loss MSELossCalculates the mean squared difference between output and target.
torch.nn.MSELoss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
3 Cross Entropy Loss CrossEntropyLossEffective for training classification problems with C categories. The optional parameter weight must be a 1D Tensor, and weights will be assigned to each category. Very effective for unbalanced training sets.In multi-class tasks, softmax activation function + cross-entropy loss function is often used, as cross-entropy describes the difference between two probability distributions. However, the neural network outputs a vector, not in the form of a probability distribution. Therefore, the softmax activation function is needed to “normalize” a vector into a probability distribution form, and then the cross-entropy loss function calculates the loss.
torch.nn.CrossEntropyLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length Cignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.reduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
4 KL Divergence Loss KLDivLossCalculates the KL divergence between input and target. KL divergence can be used to measure the distance between different continuous distributions, and is very effective for direct regression in the space of continuous output distributions (discrete sampling).
torch.nn.KLDivLoss(reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
5 Binary Cross Entropy Loss BCELossCross-entropy calculation function for binary classification tasks. Used to measure reconstruction error, such as in autoencoders. Note that the target values t[i] should be in the range of 0 to 1.
torch.nn.BCELoss(weight=None, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”
6 BCEWithLogitsLossThe BCEWithLogitsLoss function integrates the Sigmoid layer into the BCELoss class. This version is numerically more stable than using a simple Sigmoid layer and BCELoss separately, as combining these two operations into one layer allows for numerical stability using the log-sum-exp trick.
torch.nn.BCEWithLogitsLoss(weight=None, reduction='mean', pos_weight=None)Parameters:
weight (Tensor, optional) – Custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”
7 Margin Ranking Loss
torch.nn.MarginRankingLoss(margin=0.0, reduction='mean')For each instance in the mini-batch, the loss function is as follows:
Parameters:
margin: default value 0
8 Hinge Embedding Loss
torch.nn.HingeEmbeddingLoss(margin=1.0, reduction='mean')For each instance in the mini-batch, the loss function is as follows:
Parameters:
margin: default value 1
9 Multi-label Classification Loss MultiLabelMarginLoss
torch.nn.MultiLabelMarginLoss(reduction='mean')For each sample in the mini-batch, the loss is calculated as follows:
10 Smooth L1 Loss SmoothL1Loss
Also known as the Huber loss function.
torch.nn.SmoothL1Loss(reduction='mean')
Where
11 Logistic Loss for Binary Classification SoftMarginLoss
torch.nn.SoftMarginLoss(reduction='mean')
12 Multi-label One-versus-all Loss MultiLabelSoftMarginLoss
torch.nn.MultiLabelSoftMarginLoss(weight=None, reduction='mean')
13 Cosine Loss CosineEmbeddingLoss
torch.nn.CosineEmbeddingLoss(margin=0.0, reduction='mean')
Parameters:
margin: default value 0
14 Multi-class Hinge Loss MultiMarginLoss
torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, reduction='mean')
Parameters:
p=1 or 2 Default: 1margin: default value 1
15 Triplet Loss TripletMarginLoss
Similar to Siamese networks, specific example: given an A, then give B and C, to see who is more similar to A.
torch.nn.TripletMarginLoss(margin=1.0, p=2.0, eps=1e-06, swap=False, reduction='mean')
Where:
16 CTC Loss CTCLossCTC connection temporal classification loss can automatically align data that is not aligned, mainly used in training serialized data that is not previously aligned. For example, speech recognition, OCR recognition, etc.
torch.nn.CTCLoss(blank=0, reduction='mean')Parameters:
reduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
17 Negative Log Likelihood Loss NLLLossNegative log likelihood loss. Used for training classification problems with C categories.
torch.nn.NLLLoss(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length Cignore_index (int, optional) – Sets a target value that will be ignored, thus not affecting the input gradient.
18 NLLLoss2dNegative log likelihood loss for image input. It calculates the negative log likelihood loss for each pixel.
torch.nn.NLLLoss2d(weight=None, ignore_index=-100, reduction='mean')Parameters:
weight (Tensor, optional) – Custom weights for each category. Must be a Tensor of length Creduction – three values, none: no reduction; mean: returns the average of the losses; sum: returns the sum of the losses. Default: mean.
19 Poisson NLL LossNegative log likelihood loss with target values following a Poisson distribution
torch.nn.PoissonNLLLoss(log_input=True, full=False, eps=1e-08, reduction='mean')Parameters:
log_input (bool, optional) – If set to True, the loss will be calculated using the formula exp(input) – target * input. If set to False, the loss will be calculated using input – target * log(input + eps).full (bool, optional) – Whether to calculate the full loss, i.e., including the Stirling approximation term target * log(target) – target + 0.5 * log(2 * pi * target).eps (float, optional) – Default value: 1e-8
References:Summary of PyTorch loss functionshttp://www.voidcn.com/article/p-rtzqgqkz-bpg.html
Recommended Reading:
Discussing the development of pre-trained language models based on recent papers
How to evaluate the fastText algorithm proposed by the author of Word2Vec? Does deep learning have no advantage in simple tasks like text classification?
From Word2Vec to Bert, a discussion on the past and present of word embeddings (Part 1)
