Decision Tree Analysis Based on Python

Recently, I read a paper that extracted image data and compared four machine learning algorithms. These four machine learning algorithms are: Decision Trees (DT), Random Forest (RF), Support Vector Machine (SVM), and Logistic Regression (LR) classification. Today, I took some time to delve into decision tree classification, as there are many tutorials available on this topic. I will make a record here for future reference.

To shorten the length, I will first present the complete code, and then provide detailed explanations of certain parts in the middle and at the end.

Complete Code

Using the built-in wine dataset from the sklearn library for decision tree analysis.

import pandas as pd
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.metrics import mean_squared_error
import graphviz
import matplotlib.pyplot as plt

# View dataset
wine = load_wine()

print("=======Dimensions====")
print(wine.data.shape)
print("=======Features====")
print(wine.feature_names)
print("=======Targets====")
print(wine.target_names)

# Split test set and training set
# Test set accounts for 30%, training set accounts for 70%
Xtrain, Xtest, Ytrain, Ytest = train_test_split(wine.data, wine.target, test_size=0.3,
                                                random_state=42)

clf = DecisionTreeClassifier(criterion="entropy")

# Build decision tree with training data
clf = clf.fit(Xtrain, Ytrain)

# Predict test set with trained decision tree
Ypred = clf.predict(Xtest)

# Score the test set
score = clf.score(Xtest, Ytest)
print(score)
# The score is very high, indicating good model fitting

# Calculate accuracy
accuracy = accuracy_score(Ytest, Ypred)
print(f"Accuracy: {accuracy}")

# Calculate mean squared error
mse = mean_squared_error(Ytest, Ypred)
print(f"Mean Squared Error: {mse}")

# Decision tree graph
feature_names = wine.feature_names
dot_data = tree.export_graphviz(clf, out_file=None,
                                feature_names=feature_names,
                                class_names=wine.target_names,
                                filled=True, rounded=True,
                                special_characters=True)
graph = graphviz.Source(dot_data)
#graph
# Error message indicates graphviz is not added to the environment variable (not important for now)

# Pruning operation + cross-validation
# To display Chinese labels
plt.rcParams['font.sans-serif']=['SimHei']
# To display negative signs correctly
plt.rcParams['axes.unicode_minus']=False
L = []
L1 = []
# Adjust the maximum depth of the decision tree
for i in range(1, 20):
    dct = DecisionTreeClassifier(criterion='entropy',
                                 random_state=10,
                                 splitter='random',
                                 max_depth=i,
                                 min_samples_leaf=10,
                                 min_samples_split=10)
    scores = cross_val_score(dct, wine.data, wine.target,
                cv=10)
    L.append([i, scores.mean()])

# Adjust the number of training samples that each child node must contain after branching
for j in range(1, 20):
    dct1 = DecisionTreeClassifier(criterion='entropy',
                                 random_state=10,
                                 splitter='random',
                                 max_depth=3,
                                 min_samples_leaf=j,
                                 min_samples_split=10)
    scores1 = cross_val_score(dct1, wine.data, wine.target,
                cv=10)
    L1.append([j, scores1.mean()])
a = pd.DataFrame(L, columns=['max_depth', 'accuracy'])
b = pd.DataFrame(L1, columns=['min_samples_leaf', 'accuracy'])

A = [a, b]
plt.figure(figsize = (15, 5), dpi=300)
for k, v in enumerate(A):
    plt.subplot(1, 2, k+1)
    plt.plot(v.iloc[:, 0], v.accuracy, color='orange')
    plt.xticks(v.iloc[:, 0])
    plt.xlabel(v.columns[0])
    plt.ylabel('Cross-validation Accuracy')
    plt.title(f'Learning Curve of {v.columns[0]}')
plt.show()

# Calculate confusion matrix
from sklearn.metrics import confusion_matrix

conf_matrix = confusion_matrix(Ytest, Ypred)
print("Confusion Matrix:\n", conf_matrix)
# This part can be visualized as a heatmap

# Calculate kappa value and OA value
from sklearn.metrics import confusion_matrix, cohen_kappa_score
import numpy as np

# Calculate OA
OA = np.sum(np.diag(conf_matrix)) / np.sum(conf_matrix)
print(f"OA: {OA:.2f}")

# Calculate Kappa
kappa = cohen_kappa_score(Ytest, Ypred)
print(f"Kappa: {kappa:.2f}")

Brief Explanation

Decision Tree Flowchart

graph = graphviz.Source(dot_data)
graph
# This works too
graph.render("wine_DT")

Since visualization requires adding graphviz to the environment path, I didn’t bother to add it, as it’s generally not needed.

Cross-validation

cross_val_score(dct1, wine.data, wine.target, cv=10)

cv=10 indicates ten-fold cross-validation.

Kappa and OA

Regarding OA and kappa values, the article describes them as follows: Overall Accuracy (OA) is one of the most common evaluation metrics that can be used to assess a model’s performance visually. The higher the accuracy, the better the classifier. The Kappa coefficient is applied to determine whether the model’s results are consistent with the actual results: kappa = 1 indicates that the results are completely consistent; kappa ≥ 0.75 is considered satisfactory; and kappa < 0.4 is considered not ideal. This part will not be translated.

References:[1] Li M, Wang Z, Cheng T, et al. Phenotyping terminal heat stress tolerance in wheat using UAV multispectral imagery to monitor leaf stay-greenness[J]. Smart Agricultural Technology, 2025: 100996.

Wishing you all the best!