GBDT gradient lifting iterative decision tree

It is an integrated model. The base classifier adopts CART regression tree. GBDT is an algorithm to classify or regress data by using additive model and continuously reducing the residual generated in the training process.
Intuitive understanding: there is a residual between each round of prediction and the actual value. The next round will make prediction according to the residual, and finally add all predictions to get the result.
GBDT generates a weak classifier through multiple rounds of iterations, and each classifier is trained based on the residuals of the previous round of classifiers.

Intuitively understand through examples:

Assuming that the ages of the three people are 5, 6 and 7 respectively, the average age of the three people is 6, so the constant of 6 is used to predict the ages of the three people, i.e. [6, 6, 6]. The residual of each person's age = age - predicted value = [5, 6, 7] - [6, 6, 6], so the residual is [- 1, 0, 1]
Next, in order to make the model more accurate, it is necessary to reduce the residual, so establish a regression tree for the residual, and then predict the accurate residual. Assuming that the residual error of this tree is [- 0.9, 0, 0.9], sum the predicted value of the previous round with the predicted value of this round. Everyone's age = [6, 6, 6] + [- 0.9, 0, 0.9] = [5.1, 6, 6.9], which is obviously closer to the real value [5, 6, 7]. At this time, the residual error of age becomes [- 0.1, 0, 0.1], and the accuracy of prediction has been improved.

Specific ideas

Let's assume that our prediction iterates for three rounds: [5, 6, 7]

Round 1 forecast: [6, 6, 6] (average)

Round 1 residual: [- 1, 0, 1]

Round 2 prediction: [6, 6, 6] (average) + [- 0.9, 0, 0.9] (the first regression tree) = [5.1, 6, 6.9]

Round 2 residual: [- 0.1, 0, 0.1]

Round 3 prediction: [6, 6, 6] (average) + [- 0.9, 0, 0.9] (the first regression tree) + [- 0.08, 0, 0.07] (the second regression tree) = [5.02, 6, 6.97]

Residual error of the third round: [- 0.08, 0, 0.03]

The residual error is getting smaller and smaller. This prediction method is GBDT algorithm.

code implementation

1. Build test data and do simple linear regression

def create_data():
    X = []
    for i in range(100):
        x = 2 * i
        y = 3 * i
        z = i
        l = x + y + z + np.random.rand() * 10
        X.append([x,y,z,l])
    return np.array(X)
 
data = create_data()

2. Construct CART regression tree, calculate mean square error and find the best cut-off point

def calc_mse(data):
    if len(data) == 0:
        return 0
    label = data[:,-1]
    return np.var(label) * len(label)	#np.var calculation variance

def select_split(data):
    min_gini = np.inf
    best_feat = None
    best_val = None
    left = None
    right = None
    data_type = 'continuity'
    for i in range(data.shape[1]-1):
        c_set = set(data[:, i])
        for val in c_set:
            arr1,arr2 = split_data(data,i,val,data_type)
            g1 = calc_mse(arr1)		#calculation error
            g2 = calc_mse(arr2)
            # g = len(arr1) / len(data) * g1 + len(arr2) / len(data) * g2 #Gini for classification
            g = g1 + g2 # Obtain the remaining minimum mean square error
            # print(i,val,g)
            if min_gini > g:
                min_gini = g
                best_feat = i
                best_val = val
                left = arr1
                right = arr2
    return best_feat,best_val,left,right

3. Construct GBDT regression tree

def create_gbdt(dataset,n_estimators,lr):
    '''
    :param data: input data
    :param n_estimators: Number of weak classifiers
    :param lr: Learning rate, also known as residual learning rate
    :return:
    '''
    data = copy.copy(dataset)
    tree_list = []
    tree_list.append(np.mean(data[:, -1]))
    data[:, -1] = data[:, -1] - np.mean(data[:, -1])
    tree_list.append(create_tree(data))
    for i in range(1,n_estimators):
        data[:,-1] = (1 - lr) * data[:,-1] # Residual error
        tree_list.append(create_tree(data))
    return tree_list

4. Construct the prediction function and use the recursive function to construct

# Forecast single tree
def predict_one(tree,X):
    if type(tree) != dict:
        return tree
    for key in tree:
        if X[key[0]] < key[1]:
            r = tree[(key[0],key[1],'left')]
        else:
            r = tree[(key[0], key[1], 'right')]
        return predict_one(r, X)
# forecast
def predict(tree_list,X,lr):
    result = tree_list[0]
    for tree in tree_list[1:]:
        result += lr * predict_one(tree,X)
    return result

5. Test and build five trees

n_estimators = 5 
gbdt_tree = create_gbdt(data,n_estimators,0.1)
print("create gbdt:",predict(gbdt_tree,data[0,:-1]))

XGBoost decision tree algorithm based on pre sorting method

The model corresponding to XGBoost is a pile of CART trees. Continuously add trees and continuously split features to grow a tree. Adding one tree at a time is actually learning a new Han Shu f(x) to fit the residual of the last prediction. When k trees are obtained after training, the score of a sample should be predicted. In fact, according to the characteristics of the sample, each tree will fall to a corresponding leaf node, and each leaf node corresponds to a score. Finally, only the corresponding scores of each tree need to be added up to be the predicted value of the sample.

Intuitive understanding

To predict the family's preference for video games, considering that young people are more likely to like video games than old people, and men prefer video games than women, children and adults are distinguished according to age, and then men and women are distinguished by gender, and each person is graded on the preference for video games one by one, as shown in the figure below:

Two trees tree1 and tree2 are trained. Similar to the principle of GBDT before, the sum of the conclusions of the two trees is the final conclusion. Therefore, the child's prediction score is the sum of the scores of the nodes where the child falls in the two trees: 2 + 0.9 = 2.9 Grandpa's prediction score is the same: - 1 + (- 0.9) = - 1.9

code implementation

import xgboost as xgb
from xgboost import plot_importance
from matplotlib import pyplot as plt

# Set model parameters
params = {
    'booster': 'gbtree',
    'objective': 'multi:softmax',   
    'num_class': 3,     
    'gamma': 0.1,
    'max_depth': 2,
    'lambda': 2,
    'subsample': 0.7,
    'colsample_bytree': 0.7,
    'min_child_weight': 3,
    'silent': 1,
    'eta': 0.001,
    'seed': 1000,
    'nthread': 4,
}

plst = params.items()

dtrain = xgb.DMatrix(X_train, y_train)	#Using native xgboost library to read libsvm data
num_rounds = 200
model = xgb.train(plst, dtrain, num_rounds)

# Predict the test set
dtest = xgb.DMatrix(X_test)
y_pred = model.predict(dtest)

# Calculation accuracy
accuracy = accuracy_score(y_test, y_pred)
print ("Accuracy:", accuracy)
# Drawing feature importance
plot_importance(model)
plt.show();

result

Accuracy: 0.9166666666666666