Full text link:https://tecdat.cn/?p=38195

The stock market plays a significant role in economic development. Due to the high return characteristics of stocks, the stock market has attracted increasing attention from institutions and investors. However, due to the complex volatility of the stock market, it can sometimes lead to significant losses for institutions or investors. Considering the risks of the stock market, research and prediction of stock price fluctuations can help investors avoid risks. Traditional time series models like ARIMA cannot describe nonlinear time series and require meeting many conditions before modeling, resulting in limited success in stock prediction.(Click “Read the original” at the end of the article to obtain the completecode data).

Video

This article proposes a hybrid deep learning model to predict stock prices. Unlike traditional hybrid prediction models, the proposed model integrates the time series model ARIMA with neural networks in a nonlinear relationship, combining the advantages of both foundational models to improve prediction accuracy. First, the stock data is preprocessed using ARIMA, and after preprocessing with ARIMA (p = 2, q = 0, d = 1), the stock sequence is input into a neural network (NN) or XGBoost. Then, a deep learning architecture formed by a pre-training – fine-tuning framework is adopted. The pre-trained model is based on a sequence-to-sequence framework using an attention mechanism, where the attention-based CNN serves as the encoder and the bidirectional LSTM as the decoder. This model first uses convolution operations to extract deep features from the raw stock data, then employs long short-term memory networks to mine long-term time series features, and finally fine-tunes using the XGBoost model, thereby fully mining stock market information across multiple periods. The proposed attention-based CNN-LSTM and XGBoost hybrid model is referred to as AttCLX. Results indicate that this model is more effective, with relatively high prediction accuracy, helping investors or institutions make decisions to expand profits and avoid risks.

Deep Learning Based on Sequence Data

(1) Basic Feedforward Neural Network (FFNN)

In a basic feedforward neural network (FFNN), the output at the current moment is determined solely by the input at the current moment, which limits the FFNN’s ability to model time series data.

(2) Recurrent Neural Network (RNN)

In a recurrent neural network (RNN), a delay is used to retain the potential state from the previous moment, and the current potential state is determined by both the previous potential state and the current input. As shown in the figure, studies have indicated that as errors propagate over time, RNNs may experience gradient vanishing, leading to long-term dependency issues.

(3) Long Short-Term Memory (LSTM) Model

Humans can selectively remember information. Through gated activation functions, the long short-term memory (LSTM) model can selectively remember updated information and forget accumulated information.

(4) Sequence-to-Sequence (seq2seq) Model

The sequence-to-sequence (seq2seq) model uses an autoencoder (i.e., encoder-decoder architecture) to analyze time series data. The seq2seq model is constructed through an encoder-decoder architecture, enhancing LSTM’s ability to learn hidden information from noisy data. In the seq2seq model, the encoder encodes the input into a context (usually the last hidden state), which is then decoded in the decoder. In the decoder, the output from the previous moment serves as the input for the next moment.

To optimize the decoded sequence, the seq2seq model employs a beam search method. Beam search and the Viterbi algorithm in Hidden Markov Models (HMM) are both based on dynamic programming. They solve the optimal estimate of the current state based on observations and previous states, referred to as decoding or inference in HMM. Solving (p(x_{k}|y_{1:k})) and (p(x_{k}|y_{1:N})) is equivalent to the forward-backward algorithm in HMM. The distribution of (x_{k}) can yield the optimal bidirectional estimate. This is the probabilistic perspective that leads to the proposed bidirectional LSTM (which combines forward and backward).

Attention Mechanism

(1) Principle of Attention Mechanism

Humans typically focus on salient information. The attention mechanism is a deep learning technique based on the human cognitive system. For input (X=(x_{1};x_{2};;x_{N})), given a query vector (q), the attention (z = 1;2;;N) describes the index of the selected information, leading to the attention distribution.

Here, the attention score is obtained through scaled dot-product, where (d) is the dimension of the input. Assuming the input key-value pairs ((K;V)=[(k_{1};v_{1});;(k_{N};v_{N})]), for a given (q), the attention function is as follows:

(2) Multi-Head Mechanism

Typically, multiple queries (Q = [q_{1};;q_{M}]) are used to compute the attention function.

Here, “jj” indicates the concatenation operation. This is known as multi-head attention (MHA). The attention mechanism can be used to generate different weights driven by data. Here, (Q), (K), and (V) are obtained through linear transformations of (X), and (WQ), (WK), and (WV) can be dynamically adjusted.

This is known as self-attention. Similarly, the output is as follows:

Thus, after applying the scaled dot-product score, the output is as follows:

Method

(1) Preprocessing

This article proposes a hybrid deep learning model to predict stock prices. Unlike traditional hybrid prediction models, the proposed model integrates the time series model ARIMA with neural networks in a nonlinear relationship, combining the advantages of both to improve prediction accuracy. First, the stock data is preprocessed using ARIMA. Inputting the raw stock market data into ARIMA can output a new sequence that more effectively describes the state.

Results from the ADF test on the original sequence and the first-order difference sequence indicate that the original sequence is non-stationary, while the first-order difference sequence is stationary. After determining (d = 1), we need to identify (AR(p)) and (MA(q)) in ARIMA. We use the autocorrelation function (ACF) and partial autocorrelation function (PACF) to determine this. The ACF and PACF of the original sequence and the first-order difference sequence are shown in the figure.

The figure shows that when the order is 2, the PACF truncates, indicating that we should use (AR(2)); while the ACF has a long tail for any order, indicating that we should use (MA(0)). Thus, (p = 2), (q = 0).

(2) Pre-training

Next, a deep learning architecture formed by a pre-training – fine-tuning framework is adopted. The pre-trained model is based on a sequence-to-sequence framework using an attention mechanism, where the attention-based CNN serves as the encoder and the bidirectional LSTM as the decoder. This model first uses convolution operations to extract deep features from the raw stock data, then employs long short-term memory networks to mine long-term time series features.

Below is a code example for demonstrating the first-order and second-order difference:

# Calculate first-order difference
training_set['diff_1'] = training_set['close'].diff(1)
plt.figure(figsize=(10, 6))
training_set['diff_1'].plot()
plt.title('First-order Difference')
plt.xlabel('Time', fontsize=12, verticalalignment='top')
plt.ylabel('diff_1', fontsize=14, horizontalalignment='center')
plt.show()

# Calculate second-order difference
training_set['diff_2'] = training_set['diff_1'].diff(1)
plt.figure(figsize=(10, 6))
training_set['diff_2'].plot()
plt.title('Second-order Difference')
plt.xlabel('Time', fontsize=12, verticalalignment='top')
plt.ylabel('diff_2', fontsize=14, horizontalalignment='center')
plt.show()

In the above code, the first-order difference of the closing price in the stock data is calculated using the diff function and stored in training_set['diff_1'], followed by plotting the first-order difference image and setting relevant image titles, axis labels, etc. Then, the second-order difference of the first-order difference is calculated and similar plotting operations are performed. These operations help us visually observe the changes in stock data after the differencing process.

In the encoder-decoder architecture based on deep learning, seq2seq can more effectively depict the hidden information of the state, but does not satisfy the assumption of linear characteristics of stock prices. The attention-based CNN (ACNN) encoder consists of self-attention layers and CNNs, which serve as the input to the LSTM decoder block. The encoder-decoder layer describes the relationship between the current sequence and the previous sequence, as well as the relationship between the current sequence and the embeddings. The encoder still employs a multi-head mechanism. When decoding the (k)th embedding, only the (k – 1)th and previous decoding states can be seen. This multi-head mechanism is known as masked multi-head attention.

The attention-based CNN (ACNN) can capture global and local dependencies that LSTM may not be able to capture, thus enhancing the robustness of the model. In the proposed encoder-decoder framework, an ACNN-LSTM structure can be adopted. In the human cognitive system, attention typically occurs before memory. The reason ACNN can capture long-term dependencies is that it integrates multi-head self-attention and convolution. Combining LSTM and ACNN enhances structural advantages and the ability to model time series data. By integrating multi-head attention and multi-scale convolution kernels, the ACNN encoder can capture significant features that LSTM may not be able to capture, while LSTM can better depict the characteristics of time series data.

(3) Fine-tuning

After decoding, an XGBoost regressor is used to obtain the output for precise feature extraction and fine-tuning. The proposed attention-based CNN-LSTM and XGBoost hybrid model is referred to as AttCLX, as illustrated in the figure.

As a fine-tuning model, XGBoost has strong scalability and flexibility.

Click the title to view previous content

TensorFlow’s long short-term memory neural network (LSTM) in Python, exponential moving average method for predicting the stock market, and visualization.

Swipe left and right to see more

01

02

03

04

Example

Model Modification

After ARIMA preprocessing, the input to the neural network is a two-dimensional data matrix generated at certain time intervals, sized TimeWindow×Features. In the empirical study of stock prediction, features include basic stock market data (opening price, closing price, highest price, lowest price, trading volume, trading amount), while the ARIMA processed sequence and residual sequence are also concatenated as features.

We adopt a retrospective technique in time series prediction, with a retrospective count of 20, meaning (o_{t}) can be obtained from (o_{t – 1};;o_{t – 5}), indicating that the width of TimeWindow is 20. The number of layers in the long short-term memory network (LSTM) is 5, with a size of 64, and the number of training epochs is 50. By introducing dropout [13] for model training, the dropout rate is set to 0.3, with 4 heads.

The data used in this article comes from Tushare (Tushare data), which provides an open free public dataset for research on the Chinese stock market, characterized by rich data, ease of use, and convenience in implementation, making it very convenient to obtain basic market data for stocks by calling its API.

# Convert the fitted values of the ARIMA model to Series type and perform slicing
predictions_ARIMA_diff = pd.Series(model.fittedvalues, copy=True)
predictions_ARIMA_diff = predictions_ARIMA_diff[3479:]

As shown in the figure

This is the situation for ARIMA used in stock price prediction.

The figure shows the residuals and the residual density plot.

Below is the relevant content regarding ARIMA + XGBoost for stock price prediction:

# Prepare data, divide into training and testing sets
train, test = prepare_data(merge_data, n_test=180, n_in=6, n_out=1)

# Perform forward validation to obtain true values and predicted values
y, yhat = walk_forward_validation(train, test)
plt.figure(figsize=(10, 6))
plt.plot(time, y, label='Residuals')
plt.plot(time, yhat, label='Predicted Residuals')
plt.title('ARIMA + XGBoost: Residuals Prediction')

This figure shows the loss curve of the ARIMA + SingleLSTM original sequence.

The figure shows the loss curve of the ARIMA + SingleLSTM residual sequence.

This figure shows the stock price prediction results of ARIMA + SingleLSTM.

The stock price prediction results of the ARIMA + SingleLSTM model and the ARIMA + BiLSTM model are shown in the figures.

The loss curve of the proposed model is shown in the figure.

The stock price prediction results of the proposed model are shown in the figure.

Comparison with Other Methods

The evaluation metrics used include Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and (R^{2}).

The comparison of different pre-training and fine-tuning models is as follows:

Conclusion

The stock market holds an extremely important position in financial and economic development. Due to the complex volatility of the stock market, predicting stock price trends can safeguard investors’ returns. Traditional time series models like ARIMA cannot describe the nonlinear relationships in stock prediction. Given the powerful nonlinear modeling capabilities of neural networks, this article proposes a hybrid model combining an attention-based CNN-LSTM with XGBoost to predict stock prices. The model integrates the ARIMA model, attention-based convolutional neural network, long short-term memory network, and XGBoost regressor in a nonlinear relationship to improve prediction accuracy. This model can capture stock market information across multiple periods. First, stock data is preprocessed using ARIMA, and then a deep learning architecture formed by a pre-training – fine-tuning framework is adopted. The pre-trained model is based on a sequence-to-sequence framework using an attention mechanism, which first utilizes multi-scale convolution based on the attention mechanism to extract deep features from the raw stock data, then employs long short-term memory networks to mine time series features, and finally fine-tunes using the XGBoost model. Results indicate that this hybrid model is more effective, helping investors or institutions achieve the goals of expanding profits and avoiding risks.

The data and code analyzed in this article are shared in the member group, scan the QR code below to join the group!

Data Acquisition

Reply “Get Data” in the public account backend to obtain free learning materials on data analysis, machine learning, deep learning, etc.

Click the end of the article“Read the original”

to obtain the complete code data materials..

This article is selected from “Predicting Bank Stock Prices in China Using a CNN-LSTM-ARIMA Hybrid Model with Attention Mechanism | Including Data Code”.

Click the title to view previous content

R language KERAS using RNN, bidirectional RNN recursive neural networks, LSTM to analyze and predict temperature time series, IMDB movie rating sentimentPython using CNN-LSTM, ARIMA, Prophet for stock price prediction research and analysis | Including data code[Video explanation] Linear time series principle and hybrid ARIMA-LSTM neural network model for predicting stock closing price research exampleRNN recurrent neural network, LSTM long short-term memory network for long-term interest rate prediction of time seriesCombining COVID-19 stock price prediction: ARIMA, KNN, and neural network time series analysisDeep learning: Keras using neural networks for simple text classification analysis of news group dataUsing PyTorch machine learning neural network classification to predict bank customer churn modelPYTHON using LSTM long short-term memory neural network parameter optimization method to predict time series shampoo sales dataPython using Keras neural network sequence model regression fitting prediction, accuracy check, and result visualizationR language deep learning convolutional neural network (CNN) for CIFAR image classification: training and result evaluation visualizationDeep learning: Keras using neural networks for simple text classification analysis of news group dataPython using LSTM long short-term memory neural network to predict unstable rainfall time seriesR language deep learning Keras recurrent neural network (RNN) model to predict multi-output variable time seriesR language KERAS using RNN, bidirectional RNN recursive neural networks, LSTM to analyze and predict temperature time series, IMDB movie rating sentimentPython using Keras neural network sequence model regression fitting prediction, accuracy check, and result visualizationPython using LSTM long short-term memory neural network to predict unstable rainfall time seriesR language neural network prediction time series: multi-layer perceptron (MLP) and extreme learning machine (ELM) data analysis reportR language deep learning: predicting time series data using Keras neural network regression modelMatlab using deep learning long short-term memory (LSTM) neural network for text data classificationR language KERAS deep learning CNN convolutional neural network classification recognition of handwritten digit image data (MNIST)MATLAB using BP neural network to predict human body fat percentage dataPython using PyTorch machine learning neural network classification to predict bank customer churn modelR language implementing CNN (convolutional neural network) model for regression data analysisSAS using the iris dataset to train an artificial neural network (ANN) model[Video] R language implementing CNN (convolutional neural network) model for regression data analysisPython using neural networks for simple text classificationR language using neural networks to improve the Nelson-Siegel model fitting yield curve analysisR language based on recurrent neural network RNN for temperature time series predictionR language neural network model predicting vehicle count time seriesR language BP neural network model analyzing student performanceMatlab using long short-term memory (LSTM) neural network for sequence data classificationR language implementing fitting neural network prediction and result visualizationUsing R language to implement neural network prediction stock examplesUsing PYTHON KERAS LSTM recursive neural network for time series predictionPython for NLP seq2seq model example: implementing neural network machine translation with KerasFor NLP in Python: using Keras multi-label text LSTM neural network classification