Full text link: tecdat.cn/?p=43689
Analyst: Bingyi Yan
When analyzing the new energy vehicle market, have you encountered the following problem: using ARIMA fails to capture sudden fluctuations in sales, while switching to LSTM easily overlooks long-term growth trends? A single model often struggles to balance between “linear” and “non-linear” aspects.(Click the end of the article “Read the original text” to get the complete intelligent agent, code, data, and documentation)..
However, in our recent project on forecasting new energy vehicle sales in Guangzhou, we combined ARIMA and LSTM, resulting in a surprising outcome—forecasting error was reduced from 18% to 10%! Today, we will break down the underlying logic of this combined model, with the original link including code and data details.
First, let’s discuss: why is a single model insufficient?
The sales of new energy vehicles are quite “tricky”. There is steady growth driven by policies (linear trend), but also sudden impacts from factors like the speed of charging pile construction and market attention (non-linear fluctuations).
- Pure ARIMA: like an “old scholar”, good at capturing overall growth trends, but slow to react to unexpected changes like a sudden increase in charging piles or a spike in Baidu index, with a forecasting error of 18.35%;
- Pure LSTM: like a “clever kid”, capable of capturing non-linear relationships, but can easily “forget its roots”. When used alone with the Baidu index, the error is 16.01%, and when adding charging pile data, it drops to 14.06%, but still not as strong as the combination. So we thought: can we let the “old scholar” stabilize the fundamentals while the “clever kid” handles sudden situations?
The underlying logic of the combined model: 1+1>2
The integration of ARIMA and LSTM is not a simple concatenation, but rather a division of labor:
- ARIMA goes first: handling the linear trend of sales data, calculating the forecast value, and passing the “unaccounted parts” (residuals) to LSTM;
- LSTM takes over: using the residuals along with auxiliary information like the number of charging piles and Baidu index to complete the non-linear fluctuations. It’s like first drawing a smooth growth line and then filling in the details of the fluctuations, so it’s not surprising that the error drops to 10.01%.
Step-by-step code breakdown: key steps from data to prediction
Data preparation: which factors are truly useful?
We analyzed four types of data (from June 2017 to February 2024):
- Sales of new energy vehicles in Guangzhou (China Passenger Car Association)
- Number of public charging piles in Guangdong Province (Charging Alliance)
- Baidu index for new energy vehicles in Guangdong Province (Baidu Index)
- Oil prices in Guangdong Province (Eastmoney) We filtered using Spearman correlation analysis:
- Charging piles and sales correlation coefficient 0.798 (highly positively correlated)
- Baidu index 0.655 (strongly positively correlated)
- Oil prices only 0.162 (almost no relation) So we decisively kept the first two as “auxiliary variables”.
Related articles
Quantitative Investment in Financial Technology: Fusion Strategies of LSTM, Wavenet, and LightGBM
Original link: tecdat.cn/?p=37184
Analyzing the advantages and disadvantages of LSTM, Wavenet, and LightGBM in financial time series forecasting, proposing a dynamic fusion framework to capture market non-linear features, applied to stock return prediction.
Building the ARIMA model: first tackle the linear trend
The core of ARIMA is to stabilize the data, we used the ADF test to measure:
- Original sales data: p=0.967 (unstable)
- After second-order differencing: p<0.001 (stable!) Finally, we chose ARIMA(4,1,0), and the fitting effect is as follows:
# Key code snippet
from statsmodels.tsa.arima.model import ARIMA
# Read data
sales_data = pd.read_csv('guangzhou_ev_sales.csv', parse_dates=['date'], index_col='date')
# Second-order differencing to stabilize the data
sales_diff2 = sales_data['sales'].diff(2).dropna()
# Build model
model_arima = ARIMA(sales_data['sales'], order=(4, 1, 0))
result_arima = model_arima.fit()
Building the LSTM model: capturing non-linear fluctuations
The “memory function” of LSTM is suitable for handling complex fluctuations, we used data from the past 6 months (sales + charging piles + Baidu index) to predict the next month:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout
# Data normalization
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(sales_data[['sales', 'charging_piles', 'baidu_index']])
# Build model
model_lstm = Sequential()
model_lstm.add(LSTM(64, return_sequences=True, input_shape=(6, 3))) # 6 months of data, 3 features
model_lstm.add(LSTM(10, return_sequences=False))
model_lstm.add(Dropout(0.2)) # Prevent overfitting
model_lstm.add(Dense(1))
Using only the Baidu index resulted in an error of 16.01%, adding charging piles reduced it to 14.88%, and combining both brought it down to 14.06%, proving that “many hands make light work”.
Combined model: the “king bomb” effect of ARIMA+LSTM
Feeding the residuals from ARIMA (the unaccounted parts) into LSTM is equivalent to “filling in the gaps”:
# Take ARIMA residuals and merge with other features
arima_residuals = result_arima.resid.values.reshape(-1, 1)
combined_features = np.concatenate((scaled_data, arima_residuals), axis=1)
# Train the combined model with new features
model_combined = Sequential()
# Structure similar to LSTM, but with one more input feature (residuals)
model_combined.add(LSTM(64, return_sequences=True, input_shape=(6, 4)))
...
The result error dropped directly to 10.01%! The comparison chart is clear at a glance:
Comparison of MAPE for each model:
We also tested the robustness, varying Dropout between 0.1-0.3, and the error fluctuation was less than 3%, indicating the model is stable.
Practical suggestions for the Guangzhou new energy vehicle market
From the model results, the number of charging piles and market attention (Baidu index) have a significant impact on sales, we can do the following:
- Build charging piles quickly: Place more in residential areas and shopping malls, as data shows this is the most direct “sales booster”;
- Promotions should focus on the Baidu index: If the index rises after an event, it indicates that the effect is in place;
- Policies need to be stable: Avoid fluctuations, long-term support is necessary to instill confidence in the market. Next time, we could try adding factors like policy documents and breakthroughs in battery technology, which might further reduce errors. (Complete code and data have been shared in the community, join the group to get it directly!)
About the Analyst
We sincerely thank Bingyi Yan for her contributions to this article. She completed her studies in Software Engineering at Guangzhou University and is currently a data analyst at China Telecom Guangzhou Branch. She specializes in Python, deep learning, mathematical modeling, and data processing. Bingyi Yan possesses solid expertise in data processing and analysis, particularly in applying deep learning techniques to solve practical business problems, providing scientific support for industry analysis through mathematical modeling and data mining.
The complete intelligent agent, data, code, and documentation are shared in the member group, scan the QR code below to join the group!
Data Acquisition
Reply “Get Data” in the public account backend to obtain free learning materials on data analysis, machine learning, deep learning, etc.
Click the end of the article“Read the original text”
to obtain the complete intelligent agent,
code, data, and documentation.
Click the title to check previous content
Below are related articles and links about Long Short-Term Memory (LSTM), covering theory, application cases, and optimization methods, involving time series forecasting, financial analysis, meteorological forecasting, and more:
1. Optimizing Seq2Seq Sequence Model Prediction of Social Media User Check-in Spatiotemporal Trajectory Data with Multi-layer LSTM in Python
-
Article link: http://mp.weixin.qq.com/s/VCIrv8DRiXy95J_mG8J-mA
-
Core content:
-
Using Seq2Seq architecture combined with multi-layer LSTM to predict user check-in trajectories, comparing the performance of LSTM, Transformer, etc., with Seq2Seq achieving the lowest RMSE (0.086) and R² reaching 0.354.
-
Innovation: Three-layer LSTM + Sigmoid activation function to optimize hyperparameters, applied in smart transportation and business location scenarios.
2. Analysis of Driver Data Hyperparameter Tuning of LSTM Using Genetic Algorithm in Python
-
Article link: http://mp.weixin.qq.com/s/QTDpqLqXraKJVZB18Ga9EA
-
Core content:
- Using genetic algorithms to optimize LSTM hyperparameters such as hidden layer dimensions and dropout rates, improving the model’s convergence speed and accuracy in time series forecasting.
3. Multi-task Learning Prediction of Stock Data Across Multiple Industries Using LSTM in Python
-
Article link: http://mp.weixin.qq.com/s/pGM9hZsJJPozGYpW4_RFpA
-
Core content:
- Combining SMA, RSI technical indicators with LSTM multi-task architecture to simultaneously predict stock trends and prices, achieving an accuracy of 0.9728, applicable to eight major sectors including photovoltaics and technology.
4. Time Series Forecasting Analysis Using LSTM in Python – Power Load Data
-
Article link: http://mp.weixin.qq.com/s/CPBnAhm5DqjWGHnywjpJfw
-
Core content:
- LSTM predicts power load, with logarithmic format data error less than 3%, significantly improving fluctuation capture in 10-day and 50-day forecasts.
5. LSTM-XGBoost Hybrid Model for Predicting Gold and Bitcoin Prices
-
Article link: http://mp.weixin.qq.com/s/1TMOgSCPmhN-sGnHIfXfvg
-
Core content:
- Combining the time series feature extraction of LSTM with the classification ability of XGBoost to predict financial product price trends, suitable for highly volatile markets.
Further Reading
-
Cutting-edge Technology: xLSTM (sLSTM/mLSTM) enhances parallel capabilities through scalar/matrix optimization, achieving an accuracy of 83% in text classification tasks.
-
Tool Recommendation: “Chameleon Mama: LSTM Time Series Prediction Code Library” includes complete data and visualization cases.
Click the original link to obtain the complete code and dataset. If you need specific LSTM application cases in certain fields (such as healthcare, NLP), please specify your requirements.
