Improving Model Performance on Multi-Typo Correction

The paper refers to the issue of reduced model performance due to multi-typo as Contextual Typo Disturbance and labels unnecessary corrections by the model as Overcorrection. To tackle these two issues, the CRASpell model is proposed, whose structure is shown in Figure 1:

From Figure 1, we can see that CRASpell is mainly divided into two parts: the left side is the Correction Module, and the right side is the Noise Modeling Module.

1. Correction Module

In the Correction Module, the Transformer Encoder combined with the Generative Block forms a basic CSC model. The Generative Block computes and outputs a generative distribution, which can be described as:

(1)

The paper uses a copy mechanism [1-2] to address the Overcorrection issue, where for each token in the input sequence, there is a one-shot vector’s copy distribution, described as:

(2)

The Copy Block outputs a copy probability, calculated as follows:

(3)

Then, using the copy probability as a weight, the copy distribution and generative distribution are combined as the final output, effectively giving the input tokens an additional bias, making the model more inclined to retain the original tokens:

(4)

2. Noise Modeling Module

To address the Contextual Typo Disturbance issue, CRASpell proposes the Noise Modeling Module, which trains the model to output similar probability distributions in both original and noisy contexts.

The structure of the Noise Modeling Module is also a Transformer Encoder combined with a Generative Block. The difference is that it has a Noisy Block at the front, which inserts noise into the typo context through substitution. The quality of the noisy context is influenced by two factors:

a. Position of Noise Insertion

The authors decide to choose within a few characters around the typo based on the experimental results in Table 3, as shown in Figure 2. If there are no errors in the sentence or if the chosen position happens to be a typo, no noise is inserted.

Figure 2

b. What Characters to Replace With

The paper uses publicly available confusion sets [3] to replace the characters at the chosen positions with similar characters. This is also a common method for generating pseudo-data in the CSC task, with replacement ratios for different types of characters being: 70% phonetic characters, 15% graphical characters, and 15% randomly selected characters.

The original input, after noise insertion by the Noisy Block, produces a new input. The output after the Noise Modeling Module is , and finally, the KL divergence loss is calculated with the generative distribution from the Correction Module:

(5)

In the Correction Module, given the training sample (X, Y), for the in equation (4), the loss for each token at each position is:

(6)

Combining equations (5) and (6), the overall loss function for model training is:

(7)

（8）

Here we can see that the Noise Modeling Module is only used during training, and the positions of noise insertion are not included in the loss calculation. This is done to avoid changing the number of typos in the training data, allowing the inserted noise to only serve to modify the contextual information.

Regarding datasets, the training set in the paper includes 10K manually annotated SIGHAN data [3-5], along with 271K automatically generated data [6]. The test set uses the SIGHAN15 test set. Additionally, the authors extracted all multi-typo data from the SIGHAN15 test set along with an equal number of negative (typo-free) data to form a multi-typo test set, as shown in Table 4. Metrics used include character-level precision, recall, and F1 scores, where predicting a typo label counts as one correct prediction, better reflecting the improvement of the proposed method on multi-typo.

Table 4

Baseline Models:

• SoftMask: proposed soft-masking strategy to enhance BERT’s error detection performance;

• SpellGCN: combines GCN with BERT to model relationships between characters;

• Tail2Tail: a BERT-based model, but replaces the decoder with CRF;

• cBERT: a work published by the authors in 2021, pre-trained BERT on CSC data; the proposed CRASpell model is also initialized with cBERT;

• PLOME: similar to cBERT but additionally integrates phonetic and graphical features obtained from pinyin and strokes;

• cBERTCopy: applies the copy mechanism on cBERT;

• cBERTNoise: applies the Noise Modeling Module on cBERT;

• cBERTRdrop: Rdrop method implemented based on cBERT [7].

1. Main Results

Table 5

From Table 5, we can see that both the Noise Modeling Module and the copy mechanism can enhance model performance. The cBERTNoise and CRASpell models that utilize the Noise Modeling Module achieved better results on the multi-typo test set than other methods (the Correction-level Precision is higher than Detection-level because its denominator is the number of predicted labels within the ground-truth range, not the total number of predicted labels). Additionally, the CRASpell model, which also employs the copy mechanism, achieved the best results on both test sets. The authors also filtered a multi-typo test set from the SIGHAN14 test set, with results shown in Table 6.

Table 6

2. Effects of Different Replaced Positions

The paper compares two methods for selecting noise insertion positions in the input sentence:

• Randomly selecting throughout the entire sentence

• Selecting near the typo

The results are shown in Table 7. The authors also inserted noise into the test set data to examine the impact of the distance between noise and typo on results, illustrated in Figure 3. Both experimental results indicate that noise closer to typos significantly affects model performance.

Table 7

Figure 3

3. Effects of Different Replaced Characters & the Copy Block

Table 8 shows the results of two noise insertion methods: randomly selecting from a dictionary and selecting from a confusion set. It can be seen that the confusion set method performs better because the phonetic and graphical characters selected from the confusion set are more representative of actual typos. The Copy Block is used to reduce BERT’s modifications to valid characters, and Table 9 shows its performance improvement on the BERT model. Since cBERT has been pre-trained on CSC data, the performance improvement from the Copy Block is relatively small.

Table 8

Table 9

4. Comparison of Different Methods for Multi-typo Texts

To demonstrate the improvement of the Noise Modeling Module on the model, the authors also implemented two cBERT-based methods:

• MultiRound: uses cBERT for multiple rounds of predictions until no further modifications occur;

• NoiseTrain: trains cBERT using data generated by the Noise Block, with noise insertion positions also contributing to loss calculations.

The results are shown in Table 10, indicating that NoiseTrain has the worst improvement effect. The authors speculate that the inserted noise increases the number of typos in the training data and lowers their quality, leading to worse performance on single-typo and zero-typo data. In contrast, the noise inserted in the Noise Modeling Module serves only as context and does not participate in loss calculations, which the authors believe is the key reason for the significant differences in results.

Table 10

To address the two limitations of previous CSC models: Contextual Typo Disturbance and Overcorrection, this paper proposes a new spelling correction model. For the first issue, the paper introduces the Noise Modeling Module, which generates noisy contexts during training, effectively enhancing the model’s correction performance on multi-typo texts. For the Overcorrection issue, the paper combines the Copy Block with the CSC model, training the model to minimize modifications when the original characters are valid. Ultimately, this method achieved a new state-of-the-art (SOTA) on the SIGHAN15 task.

Organizing is not easy, please give a thumbs up!