Task07
This study refers to the Datawhale open source learning: https://github.com/datawhalechina/machine-learning-toy-code/tree/main/ml-with-sklearn
The content is arranged as follows, mainly some code implementation and some principles.

7. Integrated Learning

In the previous chapter, we talked about the decline in the effect of the dimension disaster mapping model. In addition to using the dimension reduction method, there is also a subspace method commonly used to deal with such high-dimensional problems. Integration is one of the common methods in subspace thinking, which combines the outputs of several algorithms or base detectors that perform well in subspace. Integrated learning accomplishes learning tasks by building and combining multiple learners, sometimes referred to as multi-classifier systems. The general strategy of integrated learning is to produce a set of "individual learners" and then combine them with a certain strategy.

The simplest method of integration is voting. Voting method is an integrated learning model that follows the principle of minority obeying majority, and reduces variance through the integration of multiple models, thereby improving the robustness of the model. Ideally, the voting method should be better than any base model.

Voting methods can be used in both regression and classification models:

• Regression voting: The predictions are the average of all model predictions.
• Classified voting: The predicted results are those that appear most frequently for all model types.

Classified voting can also be divided into hard and soft votes:

• Hard votes: The class with the most predicted results for all votes.
• Soft votes: The predicted result is the class with the highest probability of summing all the voting results.

7.1. Bagging

Unlike voting, Bagging not only integrates the final predictions of the model, but also uses strategies to influence the training of the base model.

At the heart of Bagging is the concept of bootstrap, where there is a playback to sample from a dataset, that is, the same sample may be sampled multiple times. First we randomly take a sample and put it into the sample set. Then we put the sample back into the initial data set and repeat the sample K times. Finally we can get a sample set of K size. In the same way, we can sample T sample sets with K samples and then train a base learner based on each sample set, a total of T learners. Combining these basic learners is the basic process of Bagging.

Sklearn provides us with an API for two Bagging methods, Bagging Regressor and Bagging Classifier. Explain the application with two official examples:

'''BaggingRegressor'''
from sklearn.svm import SVR
from sklearn.ensemble import BaggingRegressor
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=100, n_features=4,n_informative=2, n_targets=1,random_state=0, shuffle=False) # Generate data
regr = BaggingRegressor(base_estimator=SVR(),n_estimators=10, random_state=0).fit(X, y) # Select ten SVR models as base models
regr.predict([[0, 0, 0, 0]])

array([-2.87202411])

'''BaggingClassifier'''
from sklearn.svm import SVC
from sklearn.ensemble import BaggingClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100, n_features=4,n_informative=2, n_redundant=0,random_state=0, shuffle=False)# Generate data
clf = BaggingClassifier(base_estimator=SVC(),n_estimators=10, random_state=0).fit(X, y)# Select ten SVR models as base models
clf.predict([[0, 0, 0, 0]])

array([1])

7.2. Boosting

Bagging reduces prediction error mainly by reducing variance. Boosting thought improves the ultimate predictive effect by continuously reducing bias. In other words, Bagging averages many strong classifiers, and Boosting combines many weak classifiers into one strong classifier.

Boosting first trains a base learner from the initial training set, then adjusts the training samples according to the performance of the base learner, so that the samples made by the previous base learner errors get more attention later, and then trains the next base learner based on the adjusted sample distribution. Repeat this until the number of base learners reaches the pre-specified value T, and finally the T base learners are weighted together.

Boosting has two common Boosting styles: Adaptive Boosting and Gradient Boosting, and their variants, Xgboost, LightGBM, and Catboost.

'''AdaBoostRegressor'''
from sklearn.ensemble import AdaBoostRegressor
from sklearn.datasets import make_regression
X, y = make_regression(n_features=4, n_informative=2,random_state=0, shuffle=False)
regr = AdaBoostRegressor(random_state=0, n_estimators=100)# Generate data
regr.fit(X, y)
AdaBoostRegressor(n_estimators=100, random_state=0)# Select 100 boosting base models
regr.predict([[0, 0, 0, 0]])

array([4.79722349])

'''AdaBoostClassifier'''
from sklearn.ensemble import AdaBoostClassifier
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=1000, n_features=4,n_informative=2, n_redundant=0,random_state=0, shuffle=False)# Generate data
clf = AdaBoostClassifier(n_estimators=100, random_state=0)# Select 100 boosting base models
clf.fit(X, y)
clf.predict([[0, 0, 0, 0]])

array([1])

7.3. Blending and Staking

Stacking is not strictly an algorithm, but a model integration strategy. Stacking integration algorithm can be understood as a two-tier integration. The first tier contains several basic classifiers, which provide the predicted results to the second tier. The second tier classifiers are usually logical regression. He uses the results of one tier classifiers as features to fit the output predicted results. Blending is essentially the same as Stacking, but with one less level of cross-validation, it can be considered a simplified version of Stacking.

'''Blending'''
import numpy as np
from sklearn import datasets 
from sklearn.model_selection import train_test_split
data, target = make_blobs(n_samples=10000, centers=2, random_state=1, cluster_std=1.0 )
## Create training and test sets
X_train1,X_test,y_train1,y_test = train_test_split(data, target, test_size=0.2, random_state=1)
## Create training and validation sets
X_train,X_val,y_train,y_val = train_test_split(X_train1, y_train1, test_size=0.3, random_state=1)

from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

# Setting the first layer classifier
clfs = [SVC(probability = True),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),KNeighborsClassifier()]

# Setting the second layer classifier
from sklearn.linear_model import LinearRegression
lr = LinearRegression()

# Output Layer 1 Verification Set Results and Test Set Results
val_features = np.zeros((X_val.shape[0],len(clfs)))  # Construct validation data prediction matrix, store validation data prediction results for each model, each column being a validation set result for model initialization
test_features = np.zeros((X_test.shape[0],len(clfs)))  # Construct test data prediction matrix, store test data prediction results for each model, each column being a model

for i,clf in enumerate(clfs):
    clf.fit(X_train,y_train) # Training model
    val_feature = clf.predict_proba(X_val)[:, 1] # Get the results of each base model in the validation set
    test_feature = clf.predict_proba(X_test)[:,1] # Obtain the results of each base model in the test set
    val_features[:,i] = val_feature # Record the results of each base model in the validation set
    test_features[:,i] = test_feature # Record the results of each base model in the test set

# Enter the results of the first level validation set into the second level training second level classifier
lr.fit(val_features,y_val)
# Output Test Data Prediction Results
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X_test, y_test, cv=3, scoring='accuracy')
print(scores.mean())

1.0

The Blending process can be represented by the following diagram, which combines code to understand the process:

'''Stacking'''
import numpy as np
from sklearn import datasets 
from sklearn.model_selection import train_test_split
data, target = make_blobs(n_samples=10000, centers=2, random_state=1, cluster_std=1.0 )
## Create training and test sets
X_train1,X_test,y_train1,y_test = train_test_split(data, target, test_size=0.2, random_state=1)
# ## Create training and validation sets
# X_train,X_val,y_train,y_val = train_test_split(X_train1, y_train1, test_size=0.3, random_state=1)

from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

# Setting the first layer classifier
clfs = [SVC(probability = True),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),KNeighborsClassifier()]

# Setting the second layer classifier
from sklearn.linear_model import LinearRegression
lr = LinearRegression()

# Output Layer 1 Verification Set Results and Test Set Results
val_features = np.zeros((X_train1.shape[0],len(clfs)))  # Construct validation data prediction matrix, store validation data prediction results for each model, each column being a validation set result for model initialization
test_features = np.zeros((X_test.shape[0],len(clfs)))  # Construct test data prediction matrix, store test data prediction results for each model, each column being a model

from sklearn.model_selection import StratifiedKFold
n_splits = 5
skf = StratifiedKFold(n_splits)
skf = skf.split(X_train1,y_train1) #Divide training data into 5 folds

for j, clf in enumerate(clfs): # Jth Model
    #Train each single model in turn
    test_features_j = np.zeros((X_test.shape[0], 5)) # Construct validation data prediction matrix (each fold), store validation data single model prediction results for each fold, and validation data training model for each column
    for i, (train, val) in enumerate(skf): # i-fold, 5-fold total
        #5-Fold cross-training uses part i I as the prediction, and the rest to train the model to obtain its predicted output as the new feature of part i I.
        X_train, y_train, X_val, y_val = X_train1[train], y_train1[train], X_train1[val], y_train1[val] # Divide training data into training data and validation data, 5 fold cross validation
        clf.fit(X_train, y_train) # Training model
        y_submission = clf.predict(X_val) # Prediction validation data. Record prediction type in y_submission
        val_features[val, j] = y_submission # The predicted results of the j th model fold i validation data (validation data comes from training data, which accounts for 1/5 of training data) are put into the training model prediction matrix according to the index of the 5-fold split
        test_features_j[:, i] = clf.predict(X_test) # Place the predicted results of the j th model fold i test data into the fold test data prediction matrix

# Enter the results of the first level validation set into the second level training second level classifier
lr.fit(val_features,y_train1)
# Output Test Data Prediction Results
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X_test, y_test, cv=3, scoring='accuracy')
print(scores.mean())

1.0

The Stacking process can be represented in the following diagram, which combines code to understand the process:

In addition, we can implement the stacking model directly with the mlxtend toolkit (pip install mlxtend).

'''Stacking(Use mlxtedn Toolkit implementation)'''
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
data, target = make_blobs(n_samples=10000, centers=2, random_state=1, cluster_std=1.0 )
## Create training and test sets
X_train1,X_test,y_train1,y_test = train_test_split(data, target, test_size=0.2, random_state=1)
# ## Create training and validation sets
# X_train,X_val,y_train,y_val = train_test_split(X_train1, y_train1, test_size=0.3, random_state=1)

from mlxtend.classifier import StackingCVClassifier
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

# Setting the first layer classifier
clfs = [SVC(probability = True),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),KNeighborsClassifier()]

# Setting the second layer classifier
from sklearn.linear_model import LinearRegression
lr = LinearRegression()

sclf = StackingCVClassifier(classifiers=clfs,  # Layer 1 Classifier
                            meta_classifier=lr,   # Layer 2 Classifier
                            random_state=42)

# Output Test Data Prediction Results
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clfs[0], X_test, y_test, cv=3, scoring='accuracy')
print(scores.mean())

1.0