1. Introduction of Apriori correlation analysis
This part can be seen in my last blog post, which mainly introduces the principle of correlation analysis.
Link: Relevance analysis of python machine learning (Apriori).
2. Case background and analysis process
There are many kinds of modern goods, and customers often struggle with what to buy. Especially for customers with selection difficulties, it is even more difficult to choose goods. Complicated shopping often brings customers a tired shopping experience. For some commodities, such as bread and milk, potato chips and coke, customers often buy things at the same time. When these things are very far away, they will reduce customers' desire to buy. Therefore, in order to obtain the maximum sales profit, we need to know what kind of goods to sell, what kind of promotion means to use, how to place the goods on the shelf and understand customers' purchase habits and preferences, which are particularly important for sellers.
Analysis purpose:
- Construct the Apriori model of commodities and analyze the correlation between commodities.
- The sales strategy is given according to the model results.
The specific steps are as follows:
- View the form of the original data.
- Preprocess the original data and convert the data form to meet the requirements of Apriori correlation analysis.
- Establish Apriori model and adjust super parameters.
- Analyze the model results. Provide sales advice.
3. Preliminary data processing
We should have a general understanding of the data of this case. There are 9835 shopping basket data and two tables in this case. It mainly includes three attributes:
| Table name | Attribute value | describe |
|---|---|---|
| Goods Order | id | The number of the category to which the commodity belongs |
| Goods | Specific commodity name | |
| Goods Types | Goods | Specific commodity name |
| Type | Commodity category |
#This file is the main file of Apriori model
import pandas as pd
file_path=open("....Fill in the file path here//GoodsOrder.csv")
data=pd.read_csv(file_path)
data.info()
des=pd.DataFrame(data.describe()).T
print(des)#View information description
There are 43367 observed values in total, and there is no missing value. The maximum id is 9835, indicating that there are 9835 shopping basket information in total.
View the proportion of sales volume:
#Proportion of sales volume
data_nums = data.shape[0]
for idnex, row in sorted[:10].iterrows():#iterrows is pd's row index and data equivalent to enumerate pd by number of times
print(row['Goods'],row['id'],row['id']/data_nums)
4. Further analysis of commodities
In step 3, we analyzed some specific commodities. Such classification has some difficulties in the implementation of follow-up decisions. We should connect these specific commodities with their own categories. Through the data in the Good Types table, we find out the category of the corresponding commodity, which is equivalent to the association in the database.
#Sales volume and proportion of various categories of goods
import pandas as pd
inputfile1 =open("C://Users / / administrator / / desktop / / Python code / / Python data analysis and mining (2nd Edition) source data and code / / Python data analysis and mining (2nd Edition) source data and code - chapters / / Chapter8 / / demo / / data / / goodorder csv")
inputfile2 =open("C://Users / / administrator / / desktop / / Python code / / Python data analysis and mining (2nd Edition) source data and code / / Python data analysis and mining (2nd Edition) source data and code - chapters / / Chapter8 / / demo / / data / / goodstypes csv")
data = pd.read_csv(inputfile1)
types = pd.read_csv(inputfile2) # read in data
group = data.groupby(['Goods']).count().reset_index()
sort = group.sort_values('id',ascending = False).reset_index()
data_nums = data.shape[0] # total
#print(sort)
del sort['index']
sort_links = pd.merge(sort,types) # Merge two datanames according to the same type in the two tables
# Sum according to categories, the total amount of each commodity category, and sort
sort_link = sort_links.groupby(['Types']).sum().reset_index()
sort_link = sort_link.sort_values('id',ascending = False).reset_index()
del sort_link['index'] # Delete the "index" column
# Calculate the percentage, then replace the column name, and finally output to the file
sort_link['count'] = sort_link.apply(lambda line: line['id']/data_nums,axis=1)
sort_link.rename(columns = {'count':'percent'},inplace = True)
print('Sales volume and proportion of various categories of goods:\n',sort_link)
# Draw a pie chart to show the proportion of sales of each kind of goods
import matplotlib.pyplot as plt
data = sort_link['percent']
labels = sort_link['Types']
plt.figure(figsize=(8, 6)) # Set canvas size
plt.pie(data,labels=labels,autopct='%1.2f%%')
plt.rcParams['font.sans-serif'] = 'SimHei'
plt.title('Proportion of sales volume of each type of commodity') # Set title
#plt.savefig('../tmp/persent.png') # Put the picture in Save in png format
plt.show()
After the specific classification of each commodity, the following general classification table is obtained.
According to our analysis, the proportion of non-alcoholic beverages is the highest, accounting for 17.51%, which is the sales force. Therefore, we analyze the commodity structure of non-alcoholic beverages.
selected = sort_links.loc[sort_links['Types'] == 'Non-alcoholic Drinks'] # Select the commodity category as "non-alcoholic beverage" and sort it
child_nums = selected['id'].sum() # Sum all "non-alcoholic beverages"
selected['child_percent'] = selected.apply(lambda line: line['id']/child_nums,axis = 1) # Find percentage
selected.rename(columns = {'id':'count'},inplace = True)
print('Sales volume and proportion of non-alcoholic beverages:\n',selected)
# outfile2 = '../tmp/child_percent.csv'
# sort_link.to_csv(outfile2,index = False,header = True,encoding='gbk') # Output results
# Draw a pie chart to show the proportion of sales of various commodities within non-alcoholic drinks
import matplotlib.pyplot as plt
data = selected['child_percent']
labels = selected['Goods']
plt.figure(figsize = (8,6)) # Set canvas size
explode = (0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.08,0.3,0.1,0.3) # Set the gap size of each block
plt.pie(data,explode = explode,labels = labels,autopct = '%1.2f%%',
pctdistance = 1.1,labeldistance = 1.2)
plt.rcParams['font.sans-serif'] = 'SimHei'
plt.title("Proportion of sales volume of non-alcoholic beverages by commodity") # Set title
plt.axis('equal')
# plt.savefig('../tmp/child_persent.png') # Save drawing
plt.show() # Display graphics
Here we are doing data conversion. Each line of the previous data represents each commodity and the corresponding id.
import pandas as pd
inputfile =open("C://Users / / administrator / / desktop / / Python code / / Python data analysis and mining (2nd Edition) source data and code / / Python data analysis and mining (2nd Edition) source data and code - chapters / / Chapter8 / / demo / / data / / goodorder csv")
data = pd.read_csv(inputfile)
# Merge the "Goods" column according to the id and separate the Goods with ","
data['Goods'] = data['Goods'].apply(lambda x:','+x)
print(data.head(10))
data = data.groupby('id').sum().reset_index()
print(data.head(10))
# Convert data format for merged product columns
data['Goods'] = data['Goods'].apply(lambda x :[x[1:]])#Convert to array
data_list = list(data['Goods'])
# Split the product name for each element
data_translation = []
for i in data_list:
p = i[0].split(',')
data_translation.append(p)
print('First 5 elements of data conversion results:\n', data_translation[0:5])
Original data:
Converted data:
Convert all the goods bought by the customer with id 1 into a list. Similarly, customers with id 2 buy
5. Correlation analysis
5.1 steps of modeling implementation:
- First, set the minimum support and minimum confidence of Apriori and input the modeling sample data
- Apriori correlation analysis algorithm is used to analyze the modeling sample data, and the parameters set by the model are taken as the conditions and objectives
At present, there is no unified standard for how to set the minimum support and minimum confidence. Most of them are through setting the initial value, and then through continuous adjustment to obtain the association results consistent with the business. The model parameters input in this paper are: minimum support 0.02 and minimum confidence 0.35.
# -*- coding: utf-8 -*-
# Code 8-6 build association rule model
from numpy import *
def loadDataSet():
return [['a', 'c', 'e'], ['b', 'd'], ['b', 'c'], ['a', 'b', 'c', 'd'], ['a', 'b'], ['b', 'c'], ['a', 'b'],
['a', 'b', 'c', 'e'], ['a', 'b', 'c'], ['a', 'c', 'e']]
def createC1(dataSet):
C1 = []
for transaction in dataSet:
for item in transaction:
if not [item] in C1:
C1.append([item])
C1.sort()
# If the mapping is frozenset unique, you can use it to construct a dictionary
return list(map(frozenset, C1))
# From candidate K-itemset to frequent K-itemset (support calculation)
def scanD(D, Ck, minSupport):
ssCnt = {}
for tid in D: # Traversal data set
for can in Ck: # Traversal candidate
if can.issubset(tid): # Judge whether the candidate contains the items of the dataset
if not can in ssCnt:
ssCnt[can] = 1 # Exclusive set to 1
else:
ssCnt[can] += 1 # If yes, add 1 to the count
numItems = float(len(D)) # Dataset size
retList = [] # L1 initialization
supportData = {} # Record the support of each data in the candidate
for key in ssCnt:
support = ssCnt[key] / numItems # Calculate support
if support >= minSupport:
retList.insert(0, key) # Join L1 if conditions are met
supportData[key] = support
return retList, supportData
def calSupport(D, Ck, min_support):
dict_sup = {}
for i in D:
for j in Ck:
if j.issubset(i):
if not j in dict_sup:
dict_sup[j] = 1
else:
dict_sup[j] += 1
sumCount = float(len(D))
supportData = {}
relist = []
for i in dict_sup:
temp_sup = dict_sup[i] / sumCount
if temp_sup >= min_support:
relist.append(i)
# Here you can set to return all the support data (or the support data of frequent itemsets)
supportData[i] = temp_sup
return relist, supportData
# Improved pruning algorithm
def aprioriGen(Lk, k):
retList = []
lenLk = len(Lk)
for i in range(lenLk):
for j in range(i + 1, lenLk): # Pairwise traversal
L1 = list(Lk[i])[:k - 2]
L2 = list(Lk[j])[:k - 2]
L1.sort()
L2.sort()
if L1 == L2: # If the first k-1 items are equal, they can be multiplied, which can prevent duplicate items
# Prune (a1 is an element in the k-item set, and b is a subset of all its k-1 items)
a = Lk[i] | Lk[j] # a is the frozenset() set
a1 = list(a)
b = []
# Traversal takes out each element, converts it to set, removes the element from a1 in turn, and adds it to b
for q in range(len(a1)):
t = [a1[q]]
tt = frozenset(set(a1) - set(t))
b.append(tt)
t = 0
for w in b:
# When b (that is, all subsets of k-1 items) are subsets of Lk (frequent), it is retained, otherwise it is deleted.
if w in Lk:
t += 1
if t == len(b):
retList.append(b[0] | b[1])
return retList
def apriori(dataSet, minSupport=0.2):
# The first three statements are to calculate and find the frequent itemsets in a single element
C1 = createC1(dataSet)
D = list(map(set, dataSet)) # Convert to a list using list()
L1, supportData = calSupport(D, C1, minSupport)
L = [L1] # Add a list box so that 1 item set is a single element
k = 2
while (len(L[k - 2]) > 0): # Are there any candidate sets
Ck = aprioriGen(L[k - 2], k)
Lk, supK = scanD(D, Ck, minSupport) # scan DB to get Lk
supportData.update(supK) # Add the key value pair of supportk to supportData
L.append(Lk) # The last value of L is an empty set
k += 1
del L[-1] # Delete the last empty set
return L, supportData # L is a frequent itemset, a list, and 1, 2, and 3 itemsets are elements respectively
# Generate all subsets of the collection
def getSubset(fromList, toList):
for i in range(len(fromList)):
t = [fromList[i]]
tt = frozenset(set(fromList) - set(t))
if not tt in toList:
toList.append(tt)
tt = list(tt)
if len(tt) > 1:
getSubset(tt, toList)
def calcConf(freqSet, H, supportData, ruleList, minConf=0.7):
for conseq in H: #Traverse all itemsets in H and calculate their confidence values
conf = supportData[freqSet] / supportData[freqSet - conseq] # Reliability calculation, combined with support data
# Lift calculation lift = P (A & B) / P (a) * P (b)
lift = supportData[freqSet] / (supportData[conseq] * supportData[freqSet - conseq])
if conf >= minConf and lift > 1:
print(freqSet - conseq, '-->', conseq, 'Degree of support', round(supportData[freqSet], 6), 'Confidence:', round(conf, 6),
'lift The value is:', round(lift, 6))
ruleList.append((freqSet - conseq, conseq, conf))
# Generation rules
def gen_rule(L, supportData, minConf = 0.7):
bigRuleList = []
for i in range(1, len(L)): # Calculate from binomial set
for freqSet in L[i]: # freqSet is the set of all k items
# Find all non empty subsets of the three itemsets, 1 itemset, 2 itemsets, up to k-1 itemset, represented by H1, which is of list type and frozenset type,
H1 = list(freqSet)
all_subset = []
getSubset(H1, all_subset) # Generate all subsets
calcConf(freqSet, all_subset, supportData, bigRuleList, minConf)
return bigRuleList
if __name__ == '__main__':
dataSet = data_translation
L, supportData = apriori(dataSet, minSupport = 0.02)
rule = gen_rule(L, supportData, minConf = 0.35)
The results are as follows:
Let's explain the output results: take the first one:
This shows that the probability of buying both (fruit juice / vegetable juice) and (whole milk) is 36.85%, and the probability of this situation is 2.66%
The subsequent results can be interpreted in the same way
According to the results of the model, we find that most shoppers mainly buy food. With the improvement of living standards and the increase of health awareness, other vegetables, rhizome vegetables and whole milk are the daily food needs of modern families. Therefore, the probability of buying other vegetables, rhizome vegetables and whole milk at the same time is high, which is in line with people's life and health awareness.
6. Model application
The results of the above model show that when customers buy other goods, they will buy whole milk at the same time. Therefore, the mall should put the whole milk on the only way or in a conspicuous place for customers to take according to the actual situation. At the same time, customers are more likely to buy other vegetables, root vegetables, yogurt, pork, butter, local eggs and a variety of fruits. Therefore, shopping malls can consider bundling sales, or adjust the layout of goods to bring these goods as close as possible, so as to improve customers' shopping experience.
reference material
Link: Apriori algorithm for machine learning association rule analysis.
Book: python data analysis and mining practice
Experimental dish dataset: Data Baidu online disk download, extraction code 7hwd.