Financial data analysis - DataJoy bank customer purchase forecast

Posted by WildcatRudy on Fri, 04 Mar 2022 16:43:00 +0100

This forecast relates to the marketing activities of Portuguese banking institutions. These marketing activities are generally based on telephone. The customer service personnel of the bank contact the customer at least once to confirm whether the customer is willing to buy the bank's products (time deposit). The basic type of task is classified task, which is to predict whether the customer will buy the bank's products.

Related package

import pandas as pd 
import numpy as np 
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
from sklearn import ensemble, 
from sklearn import model_selection
from sklearn import multiclass
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from sklearn.metrics import roc_curve, auc,roc_auc_score
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.metrics import confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
from sklearn.model_selection import GridSearchCV
from imblearn.over_sampling import SMOTE
from scipy.stats import chi2_contingency

Data preprocessing

Data import

train_set = pd.read_csv('train_set.csv')
test_set = pd.read_csv('test_set.csv')   
#(25317, 18)
#(10852, 17) 

Data field
ID: customer unique ID
Age: age
job: position
Marital: marital status
education: education
Default: whether there is a default record
Balance: the average balance of the account every year
Housing: is there a housing loan
Loan: is there a personal loan
contact: communication with customers
Last contact date: day
Month: the month of the last contact
Duration: the communication duration of the last contact
campaign: the number of exchanges with customers in this activity
pdays: how long has it been since the last time the customer was contacted in the last activity (999 means no contact)
previous: the number of times we communicated with the customer before this activity
poutcome: the result of the last activity
y: Predict whether customers will order time deposit business

Duplicate value processing

test_set = test_set.drop_duplicates()
train_set = train_set.drop_duplicates() 
#(25317, 18)
#(10852, 17)
#The data has no duplicate values

Missing value processing

#The data has no NA value but has an unknow n value
# Work, education and communication methods are filled with modes. The result of the last communication is poutcome. This feature is abandoned because there are too many deficiencies

Exploratory analysis

#Discrete variable column name
object_columns = ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact','month']
#Continuous variable column name
num_columns = ['age', 'balance', 'duration', 'campaign', 'pdays', 'previous','day']

discrete variable

def barplot(x,y, **kwargs):
    sns.barplot(x=x , y=y)
    x = plt.xticks(rotation=45)
f = pd.melt(train_set,  value_vars=object_columns ,id_vars = 'y')
g = sns.FacetGrid(f, col="variable", col_wrap=3, sharex=False, sharey=False, size=5)
g =,"value",'y')       

Position: retirees and students are more likely to buy time deposits, followed by the unemployed and managers, and the willingness of blue collar fixed-term financial management is lower
Marital status: single people are more willing to manage money regularly than divorced and married people
Education level: the willingness of regular financial management is reduced according to the education level of University, high school and junior middle school
Whether there is a default record: people without a default record are twice as likely to buy regular financial management as those with a default record
Whether there is housing loan: people without housing loan may buy fixed-term financial management more than twice as much as those with housing loan
Whether there is personal loan: people without personal loan are more likely to buy fixed-term financial management
Communication with customers: communication by telephone and mobile phone has no obvious impact on customers' decision to purchase time deposits
The last contact month: December, October, September and March, the proportion of time deposits purchased is high

g = sns.FacetGrid(train_set, col='marital',size=5), 'default', 'y', 'education') 

Single people with no record of default and college degree are more likely to buy time deposits
People with divorce, a record of default and a college degree are less likely to buy time deposits

def barplot(x,y, **kwargs):
    sns.barplot(x=x , y=y)
plt.figure(figsize=(16, 12))
g = sns.FacetGrid(train_set, col='education',col_order=['primary','secondary','tertiary'],size=5), 'job', 'y')  

With the improvement of educational background, the willingness of retirees and blue collar workers to buy regular financial management increases, while the willingness of students to buy regular financial management decreases

continuous variable

f = pd.melt(train_set,  value_vars=num_columns ,id_vars = 'y')
g = sns.FacetGrid(f, col="variable", col_wrap=3, sharex=False, sharey=False, size=5,hue='y')
g =,"value",bins=20) 

Age: more willing to buy regular financial products before the age of 30, less willing to buy regular financial products at the age of 30-60, and significantly more likely to buy regular financial products after the age of 60
Communication duration: the rate of purchasing fixed-term financial management increased after about 300 minutes of communication
Number of communication with customers in this activity: the increase of communication times can not significantly improve the ratio of customers to buy regular financial management
How long has it been since the last time the customer was contacted in the last activity: in 140-160180-200 days, the rate of customers purchasing regular financial management has increased significantly
Number of communication with the customer before this activity: the increase of communication times can not significantly improve the ratio of customers to buy regular financial management, which may be counterproductive
Date: there will be three peak purchases of regular financial management every month, and customers least want to buy regular financial management on the 23rd-25th of each month

Characteristic Engineering

descriptive statistics

#If the data distribution is unbalanced, up sampling can be used to solve the problem of sample imbalance

#descriptive statistics 

#Distribution of data
categorical_features = ['balance','duration','campaign','pdays','previous']
f = pd.melt(train_set,  value_vars=categorical_features,id_vars=['y'])
g = sns.FacetGrid(f, col="variable",  col_wrap=3, sharex=False, sharey=False, size=5)
g =, "value") 

It can be seen that there are many extreme values of data. Deleting extreme values will have a certain impact on the source data. The data will be processed in boxes later

Characteristic structure

#View the values of category variables
for column in object_columns:
    print(column,':  ',train_set[column].unique())
#job :   ['management' 'technician' 'admin.' 'services' 'retired' 'student' 'blue-collar' 'entrepreneur' 'housemaid' 'self-employed' 'unemployed']
#marital :   ['married' 'divorced' 'single']
#education :   ['tertiary' 'primary' 'secondary']
#default :   ['no' 'yes']
#housing :   ['yes' 'no']
#loan :   ['no' 'yes']
#contact :   ['cellular' 'telephone']
#month :   ['may' 'apr' 'jul' 'jun' 'nov' 'aug' 'jan' 'feb' 'dec' 'oct' 'sep' 'mar']

#Characteristics of structural quarter and half year
def quarter(data):
    a = ''
    if data in ['jan','feb','mar']:
        a = 'Q1'
    elif data in ['apr','may','jun']:
        a = 'Q2'
    elif data in ['jul','aug','sep']:
        a = 'Q3'
        a = 'Q4'
    return a

def halfyear(data):
    a = ''
    if data in ['jan','feb','mar','apr','may','jun']:
        a = 'H1'
        a = 'H2'
    return a

The discrete data is encoded and the continuous data is divided into boxes. The methods of box division include equivalence, equal width, clustering, chi square, minimum entropy, etc. Here, qcut and cut are selectively used to divide the data according to the characteristics of the data, and the number of boxes is determined by woe and iv values.

def CalcWOE(df, col, target):
    : df dataframe
    : col Note that this column has been divided into boxes. Now calculate the weight of each box WOE And overall IV
    : target Target column 0-1 value
    : return Return each box WOE And overall IV
    total = df.groupby([col])[target].count()
    total = pd.DataFrame({'total': total})
    bad = df.groupby([col])[target].count() - df.groupby([col])[target].sum()
    bad = pd.DataFrame({'bad': bad})
    regroup = total.merge(bad, left_index=True, right_index=True, how='left')
    regroup.reset_index(level=0, inplace=True)
    N = sum(regroup['total'])
    B = sum(regroup['bad'])
    regroup['good'] = regroup['total'] - regroup['bad']
    G = N - B
    regroup['bad_pcnt'] = regroup['bad'].map(lambda x: x * 1.0 / B)
    regroup['good_pcnt'] = regroup['good'].map(lambda x: x * 1.0 / G)
    regroup['WOE'] = regroup.apply(
        lambda x: np.log(x.good_pcnt * 1.0 / x.bad_pcnt), axis=1)
    WOE_dict = regroup[[col, 'WOE']].set_index(col).to_dict(orient='index')
    IV = regroup.apply(
        lambda x:
        (x.good_pcnt - x.bad_pcnt) * np.log(x.good_pcnt * 1.0 / x.bad_pcnt),
    IV_SUM = sum(IV)
    return {'WOE': WOE_dict, 'IV_sum': IV_SUM, 'IV': IV}

#Judge whether woe satisfies monotonicity
def BadRateMonotone(df, sortByVar, target):
    #df[sortByVar] this column has been divided into boxes
    df2 = df.sort_values(by=[sortByVar])
    total = df2.groupby([sortByVar])[target].count()
    total = pd.DataFrame({'total': total})
    bad = df2.groupby([sortByVar])[target].count() - df2.groupby(
    bad = pd.DataFrame({'bad': bad})
    regroup = total.merge(bad, left_index=True, right_index=True, how='left')
    regroup.reset_index(level=0, inplace=True)
    combined = zip(regroup['total'], regroup['bad'])
    badRate = [x[1] * 1.0 / x[0] for x in combined]
    badRateMonotone = [
        badRate[i] < badRate[i + 1] for i in range(len(badRate) - 1)
    Monotone = len(set(badRateMonotone))
    if Monotone == 1:
        return True
        return False

def num_band(df, columns, target, min_num, max_num):
    result = []
    for col in columns:
        for i in range(min_num, max_num):
                df['band'] = pd.cut(df[col], i)
                WOE_IV = CalcWOE(df, 'band', target)
                T_F = BadRateMonotone(df, 'band', target)
                result.append([col, i, WOE_IV['IV_sum'], T_F])
    return pd.DataFrame(result, columns=['column', 'num', 'IV_sum', 'T_F'])

num_band(train_set, num_columns, 'y', 2, 10)
#Select woe the number of bins with monotonicity and maximum IV value
for dataset in [train_set]:
    dataset['balanceBin'] = pd.qcut(dataset['balance'], 5)
    dataset['ageBin'] = pd.cut(dataset['age'].astype(int), [0, 30, 60, 100])
    dataset['quarter'] = dataset['month'].map(quarter)
    dataset['halfyear'] = dataset['month'].map(halfyear)
    dataset['dayBin'] = pd.cut(dataset['day'], 2)
    dataset['durationBin'] = pd.qcut(dataset['duration'], 9)
    dataset['campaignBin'] = pd.qcut(dataset['campaign'], 2)
    dataset['pdaysBin'] = pd.cut(dataset['pdays'], 9)
    dataset['previousBin'] = pd.cut(dataset['previous'], 9)
    dataset['all_previous'] = dataset['campaign'] + dataset['previous']
    dataset['all_previousBin'] = pd.cut(dataset['all_previous'], 2)

Feature screening

Judge the maximum proportion of each category (box division result) in the overall proportion. If the proportion of a category exceeds 95%, it indicates that this feature is seriously skewed, and the feature should be removed. The final result is to remove previousBin and all_previousBin feature

def MaximumBinPcnt(df, col):
    N = df.shape[0]
    total = df.groupby([col])[col].count()
    pcnt = total * 1.0 / N
    return max(pcnt)

discrete_columns = [
    'job', 'marital', 'education', 'default', 'housing', 'loan', 'contact',
    'month', 'y', 'balanceBin', 'ageBin', 'quarter', 'halfyear', 'dayBin',
    'durationBin', 'campaignBin', 'pdaysBin', 'previousBin', 'all_previousBin'

for column in discrete_columns:
    print(column, ':', MaximumBinPcnt(train_set, column))

Encode data labels

label = LabelEncoder()
for dataset in [train_set]:
    dataset['job_Code'] = label.fit_transform(dataset['job'])
    dataset['ageBin_Code'] = label.fit_transform(dataset['ageBin'])
    dataset['marital_Code'] = label.fit_transform(dataset['marital'])
    dataset['education_Code'] = label.fit_transform(dataset['education'])
    dataset['default_Code'] = label.fit_transform(dataset['default'])
    dataset['balanceBin_Code'] = label.fit_transform(dataset['balanceBin'])
    dataset['housing_Code'] = label.fit_transform(dataset['housing'])
    dataset['loan_Code'] = label.fit_transform(dataset['loan'])
    dataset['contact_Code'] = label.fit_transform(dataset['contact'])
    dataset['dayBin_Code'] = label.fit_transform(dataset['dayBin'])
    dataset['month_Code'] = label.fit_transform(dataset['month'])
    dataset['durationBin_Code'] = label.fit_transform(dataset['durationBin'])
    dataset['campaignBin_Code'] = label.fit_transform(dataset['campaignBin'])
    dataset['pdaysBin_Code'] = label.fit_transform(dataset['pdaysBin'])
    dataset['quarter_Code'] = label.fit_transform(dataset['quarter'])
    dataset['halfyear_Code'] = label.fit_transform(dataset['halfyear'])

Use chi square test to screen discrete variables,

for col in columns_train_data_x:
    obs = pd.crosstab(train_set['y'],
    chi2, p, dof, expect = chi2_contingency(obs)
    print("{} Chi square test p value: {:.4f}".format(col, p))
#The P value of chi square test for all features was significantly less than 0.01, and all features were retained

columns_train_data_x = [
    'job_Code', 'ageBin_Code', 'marital_Code', 'education_Code',
    'default_Code', 'balanceBin_Code', 'housing_Code', 'loan_Code',
    'contact_Code', 'dayBin_Code', 'month_Code', 'durationBin_Code',
    'campaignBin_Code', 'pdaysBin_Code', 'quarter_Code', 'halfyear_Code'
Target = ['y']
#spearman is used to test the correlation between the data
def correlation_heatmap(df):
    _ , ax = plt.subplots(figsize =(20, 16))
    colormap = sns.diverging_palette(220, 10, as_cmap = True)
    _ = sns.heatmap(
        cmap = colormap,
        cbar_kws={'shrink':.9 }, 
        linewidths=0.1,vmax=1.0, linecolor='white',
        annot_kws={'fontsize':10 }
    plt.title('Pearson Correlation of Features', y=1.05, size=15)


quarter_Code and halfyear_Code is highly relevant, so only halfyear is selected_ Code

Modeling analysis

Training data

columns_train_data_x = [
    'job_Code', 'ageBin_Code', 'marital_Code', 'education_Code',
    'default_Code', 'balanceBin_Code', 'housing_Code', 'loan_Code',
    'contact_Code', 'dayBin_Code', 'month_Code', 'durationBin_Code',
    'campaignBin_Code', 'pdaysBin_Code', 'halfyear_Code'
train_data_x = train_set[columns_train_data_x]
train_data_y = train_set['y']
train_data_x = pd.get_dummies(train_data_x , columns=columns_train_data_x)

The samples are up sampled by BorderlineSMOTE, ADASYN, SMOTETomek, etc

#Oversampling samples
train_data_x,train_data_y = SMOTE().fit_resample(train_data_x,train_data_y)
#0    22356
#1    22356


The full name of GBDT is Gradient Boosting Decision Tree, which is an integrated learning algorithm and is widely used in industry.
The grid search and optimization of GBDT can be carried out directly. If the amount of data is too large, the greedy algorithm can be used for optimization, but the greedy algorithm can only achieve local optimization.

GBDT_n_estimators = [120, 300]
GBDT_learning_rate = [0.001, 0.01]
GBDT_max_features = ['sqrt']
GBDT_max_depth = [3, 5, 8]
#GBDT_min_samples_split = [1, 2, 5, 10, 15, 100]
#GBDT_min_samples_leaf = [1, 2, 5, 10]
#GBDT_subsample = [0.5, 0.6, 0.7, 0.8, 0.9, 1]

param_grid = {
    'n_estimators': GBDT_n_estimators,
    'learning_rate': GBDT_learning_rate,
    'max_features': GBDT_max_features,
    'max_depth': GBDT_max_depth,
    #'min_samples_split': GBDT_min_samples_split,
    #'min_samples_leaf': GBDT_min_samples_leaf,
    #'subsample': GBDT_subsample
cv_split = model_selection.ShuffleSplit(n_splits=10,
model_tunning = GridSearchCV(ensemble.GradientBoostingClassifier(),
                             scoring='roc_auc'), train_data_y)
print('Optimal score', model_tunning.best_score_)  #Model maximum score
print('Optimal parameters', model_tunning.best_params_)  #Optimal parameters
print('Optimal model', model_tunning.best_estimator_)  #Optimal model
best_model = model_tunning.best_estimator_

Model evaluation methods include confusion matrix, accuracy rate, recall rate, ROC curve, ACU value, etc

#Confusion matrix
x_train, x_test, y_train, y_test = model_selection.train_test_split(
    train_data_x, train_data_y, train_size=.7)
model = ensemble.GradientBoostingClassifier(n_estimators=300,
                                            subsample=1), y_train)
y_predict = model.predict(x_test)
cm = confusion_matrix(y_test, y_predict)
from sklearn.metrics import accuracy_score, precision_score,recall_score
print(accuracy_score(y_test, y_predict))  #Accuracy
print(precision_score(y_test, y_predict, average='weighted'))  #Weighted accuracy
print(recall_score(y_test, y_predict))  #recall 

K-fold cross validation, ROC was used as the evaluation method_ auc

score = cross_val_score(model, train_data_x, train_data_y, cv=5, n_jobs=1, scoring='roc_auc')
#Final model score 0.9830339716115262

ROC curve

#ROC/AUC, y_train)
def get_rocauc(y,X,clf):
    plt.plot(FPR,recall,color='red',label='ROC curve (area = %0.2f)'%area)
    plt.xlabel('False Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc='lower right')
    return threshold

threshold=get_rocauc(y_test, x_test,model)

You are welcome to communicate and criticize

Topics: Python Machine Learning Data Analysis Data Mining