1, Introduction
1.1 concept
As shown in the figure, it is a decision tree. The square represents the decision block and the ellipse represents the terminating block, indicating that the conclusion has been reached and the operation can be terminated. The left and right arrow branch es are led out from the judgment module, which can reach the next judgment module or terminate the module. A hypothetical e-mail classification system is constructed.
1.2 construction of decision tree
1.2.1 general process:
1. Collect data
2. Prepare data
3. Analysis data
4. Training algorithm
5. Test algorithm
6. Use algorithm
The pseudo code function createBranch to create a branch is as follows:
Detect whether each sub item in the dataset belongs to the same classification
If so return Class label Else Find the best features to divide the dataset Partition dataset Create branch node for Subset of each partition Call function createBranch And add the returned result to the branch node return Branch node
1.3 information gain
The change of information before and after dividing the data set is called information gain. Knowing how to calculate the information gain, we can calculate the learning gain obtained by dividing the data set by each eigenvalue. The feature with the highest information gain is the best choice.
Where n is the number of classifications
Code part for calculating Shannon entropy for a given dataset:
from math import log
import operator
def calcshannonEnt(dataSet):
nuMEntries = len(dataSet)
labelCounts = {}
for featVec in dataSet:
currentLabel = featVec[-1]
if currentLabel not in labelCounts.keys():
labelCounts[currentLabel] = 0
labelCounts[currentLabel] += 1
shannonEnt = 0.0
for key in labelCounts:
prob = float(labelCounts[key])/nuMEntries
shannonEnt -= prob * log(prob,2)
return shannonEnt
First, we need to calculate the total number of instances in the dataset. Here, we explicitly declare a variable to save the total number of instances. Then create a data dictionary whose key value is the value of the last column. If the current key value does not exist, expand the dictionary and add the current key value to the dictionary. Here, each key value records the occurrence times of the current category. Finally, the occurrence probability of all class labels is used to calculate the occurrence probability of the category. We will use this probability to calculate Shannon entropy and count the times of all class labels. Finally, the occurrence frequency of all class labels is used to calculate the probability of class occurrence.
You can use createDataSet() to get a simple authentication dataset
def createDataset():
dataSet = [[1,1,'yes'],[1,1,'yes'],[1,0,'no'],[0,1,'no'],[0,1,'no']]
labels = ['no surfacing','flippers']
return dataSet,labels
The higher the entropy, the more mixed data. We can add more categories in the data set to observe how the entropy changes. We add a third category called may to test the change of entropy:
myData[0][-1]='maybe'
print(myData)
print(calcshannonEnt(myData))1.4 data division
In addition to measuring the information entropy, the classification algorithm also needs to divide the data set and measure the entropy of the divided data set, so as to judge whether the data set is divided correctly. The information entropy is calculated once for the result of dividing the data set according to each feature, and then it is judged which feature is the best way to divide the data set.
Given feature Division:
def splitDataSet(dataSet, axis, value):
# Create a list of returned datasets
retDataSet = []
# Traversal dataset
for featVec in dataSet:
if featVec[axis] == value:
# Remove axis feature
reducedFeatVec = featVec[:axis]
# Add eligible to the returned dataset
reducedFeatVec.extend(featVec[axis + 1:])
retDataSet.append(reducedFeatVec)
# Returns the partitioned dataset
return retDataSet
def createDataSet():
# data set
dataSet = [[1, 1, 'yes'],
[1, 1, 'yes'],
[1, 0, 'no'],
[0, 1, 'no'],
[0, 1, 'no']]
# Classification properties
labels = ['no surfacing', 'flippers']
# Return dataset and classification properties
return dataSet, labels
if __name__ == '__main__':
dataSet, labels = createDataSet()
print(dataSet)
print(splitDataSet(dataSet, 0, 1))
print(splitDataSet(dataSet, 0, 0))
Select the best way to divide the data:
from math import log
def calcShannonEnt(dataSet):
# Returns the number of rows in the dataset
numEntries = len(dataSet)
# Save a dictionary of the number of occurrences of each label
labelCounts = {}
# Each group of eigenvectors is counted
for featVec in dataSet:
# Extract label information
currentLabel = featVec[-1]
# If the label is not put into the dictionary of statistical times, add it
if currentLabel not in labelCounts.keys():
labelCounts[currentLabel] = 0
# Label count
labelCounts[currentLabel] += 1
# Shannon entropy
shannonEnt = 0.0
# Calculate Shannon entropy
for key in labelCounts:
# The probability of selecting the label
prob = float(labelCounts[key])/numEntries
# Calculation of Shannon entropy by formula
shannonEnt -= prob * log(prob, 2)
# Return Shannon entropy
return shannonEnt
def splitDataSet(dataSet, axis, value):
# Create a list of returned datasets
retDataSet = []
# Traversal dataset
for featVec in dataSet:
if featVec[axis] == value:
# Remove axis feature
reducedFeatVec = featVec[:axis]
# Add eligible to the returned dataset
reducedFeatVec.extend(featVec[axis + 1:])
retDataSet.append(reducedFeatVec)
# Returns the partitioned dataset
return retDataSet
def chooseBestFeatureToSplit(dataSet):
# Number of features
numFeatures = len(dataSet[0]) - 1
# Calculate Shannon entropy of data set
baseEntropy = calcShannonEnt(dataSet)
# information gain
bestInfoGain = 0.0
# Index value of optimal feature
bestFeature = -1
# Traverse all features
for i in range(numFeatures):
# Get the i th all features of dataSet
featList = [example[i] for example in dataSet]
# Create set set {}, elements cannot be repeated
uniqueVals = set(featList)
# Empirical conditional entropy
newEntropy = 0.0
# Calculate information gain
for value in uniqueVals:
# The subset of the subDataSet after partition
subDataSet = splitDataSet(dataSet, i, value)
# Calculate the probability of subsets
prob = len(subDataSet) / float(len(dataSet))
# The empirical conditional entropy is calculated according to the formula
newEntropy += prob * calcShannonEnt(subDataSet)
# information gain
infoGain = baseEntropy - newEntropy
# Print information gain for each feature
print("The first%d The gain of each feature is%.3f" % (i, infoGain))
# Calculate information gain
if (infoGain > bestInfoGain):
# Update the information gain to find the maximum information gain
bestInfoGain = infoGain
# Record the index value of the feature with the largest information gain
bestFeature = i
# Returns the index value of the feature with the largest information gain
return bestFeature
Operation screenshot:
1.5 recursive construction of decision tree
Working principle: get the original data set, and then divide the data set based on the best attribute value. Since there may be more than two eigenvalues, there may be data set division greater than two branches. After the first division, the data will be passed down to the next node of the tree branch. On this node, we can divide the data again, Therefore, we can use the principle of recursion to deal with data sets.
Conditions for the end of recursion: the program traverses the attributes of all partitioned data sets, or all instances under each branch have the same classification. If all instances have the same classification, a leaf node or termination block is obtained, and any data reaching the leaf node must belong to the classification of the leaf node.
If the class label is still not unique after the dataset has processed all attributes, we need to decide how to define the leaf node. We usually use the majority voting method to determine the classification of the leaf node. The code is as follows:
def majorityCnt(classList):
classCount={}
for vote in classList: #Count the number of occurrences of each element in the classList
if vote not in classCount.keys():classCount[vote]=0
classCount[vote]+=1
sortedClassCount=sorted(classCount.iteritems(),key=operator.itemgetter(1),reverse=True) #Sort dictionary
return sortedClassCount[0][0]
The code for creating the tree is as follows:
def createTree(dataSet,labels):
classList = [example[-1] for example in dataSet]
if classList.count(classList[0]) == len(classList):#If the categories are exactly the same, continue to divide
return classList[0]
if len(dataSet[0]) == 1: #When all features are traversed, the category with the most occurrences is returned
return majorityCnt(classList)
bestFeat = chooseBestFeatureTOSplit(dataSet) #Select the optimal feature
bestFeatLabel = labels[bestFeat] #Optimal feature label
myTree = {bestFeatLabel:{}}
del (labels[bestFeat]) #Get all attribute values contained in the list
featValues = [example[bestFeat] for example in dataSet]
uniqueVals = set(featValues)
for value in uniqueVals: #Create decision tree
subLabels = labels[:]
myTree[bestFeatLabel][value] = createTree(splitDataSet(dataSet,bestFeat,value),subLabels)
return myTree
The variable myTree contains many nested dictionaries representing tree structure information. Starting from the left, the first keyword no surfacing is the feature name of the first partitioned dataset, and the value of this keyword is another data dictionary. The second keyword is the data set divided by the no surfacing feature. The values of these keywords are the child nodes of the no surfacing node. These values may be class labels or another data dictionary. If the value is a class label, the child node is a leaf node; If the value is another data dictionary, the child node is a judgment node. This format structure repeats continuously to form the whole tree. This tree contains three leaf nodes and two judgment nodes.
2, Drawing a tree in python using the Matplotlib annotation
Draw tree nodes with text annotations:
import matplotlib.pyplot as plt
from pylab import mpl
decisionNode = dict(boxstyle='sawtooth',fc="0.8")#Define text box and arrow formats
leafNode = dict(boxstyle="round4",fc="0.8")
arrow_args = dict(arrowstyle="<-")
def plotNode(nodeTxt,centerPt,parentPt,nodeType):#Draw annotation with arrow
createPlot.ax1.annotate(nodeTxt,xy=parentPt,xycoords='axes fraction',xytext=centerPt,textcoords='axes fraction',va="center",ha="center",bbox=nodeType,arrowprops=arrow_args)
def createPlot():
fig = plt.figure(1,facecolor='white')
fig.clf()
createPlot.ax1 = plt.subplot(111,frameon=False)
mpl.rcParams['font.sans-serif'] = ['SimHei'] # Blackbody
plotNode('Decision node',(0.5,0.1),(0.1,0.5),decisionNode)
plotNode('Leaf node',(0.8,0.1),(0.3,0.8),leafNode)
plt.show()Operation screenshot:
I haven't finished it yet. I'll write it tomorrow. 55555!!!!