Surprise (Simple Python Recommendation System Engine) is a recommendation system library, which is one of the scikit series. Simple and easy to use, while supporting a variety of recommendation algorithms (basic algorithm, collaborative filtering, matrix decomposition, etc.).

The following purposes are taken into account when designing surprise:

• Let users control their experiments perfectly. To this end, the document is emphasized in particular, trying to be as clear and accurate as possible by pointing out every detail of the algorithm.
• Reduce the pain of data set processing. Users can use built-in data sets (Jester) and their own custom data sets.
• Provide a variety of ready-to-use prediction algorithms, such as baseline algorithm, neighborhood method, matrix factorization (SVD, PMF, SVD ++, NMF) and so on. In addition, various similarity measures (cosine, MSD, Pearson...) are built in.
• New algorithm ideas can be easily realized.
• Provides tools for evaluating, analyzing and comparing the performance of algorithms. Using powerful CV iterators (inspired by excellent scikit-learn ing tools) and detailed searches for a set of parameters, cross validation programs can be run very easily.

1.Surprise installation

pip install numpy
pip install scikit-surprise

Make sure that numpy module is installed before installation.

2. Basic Algorithms

Neighbor-based method (collaborative filtering) can set different metrics.

Supporting different evaluation criteria

3.Surprise use

(1) Load the data set that comes with it

#-*- coding:utf-8 -*-
# You can use the various recommendation system algorithms mentioned above
from surprise import SVD
from surprise import Dataset, print_perf
from surprise.model_selection import cross_validate

# Default loading of movielens dataset
data = Dataset.load_builtin('ml-100k')
# K-fold cross validation (k=3), which is now discarded
# data.split(n_folds=3)
# Try SVD Matrix Decomposition
algo = SVD()
# Test the effect on the data set
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
#Output result
print_perf(perf)

Operation results:

(2) Load your own dataset

from surprise import SVD
from surprise import Dataset, print_perf, Reader
from surprise.model_selection import cross_validate
import os

# Specify the path of the file
file_path = os.path.expanduser('data.csv')
# Tell the text reader what the format of the text is
reader = Reader(line_format='user item rating', sep=',')
# Loading data
data = Dataset.load_from_file(file_path, reader=reader)
algo = SVD()
# Test the effect on the data set
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
# Output results
print_perf(perf)

Attention should be paid to:

1. Can not recognize Chinese, if there is Chinese, it needs to be converted into ID number for operation (listed below is a simple conversion method)

2. You can't have a header. You need to remove Chinese columns from the header and metadata.

3. Reader needs to be modified. line_format is the column of data, and sep is the separation mode (the initial division mode of table format is',').

A simple data conversion method:

#-*- coding:utf-8 -*-
# Constructing Item id

import pandas as pd

df = pd.read_csv('train_score.csv', encoding="gbk")
# Read the data in the second column
item_name = df.iloc[:, 1]
item = {}
item_id = []
num = 0
# Associate each different item with id number
for i in item_name:
    if i in item:
        item_id.append(item[i])
    else:
        item[i] = num
        item_id.append(num)
        num += 1
print item_id
df['itemId'] = item_id
df.to_csv("data.csv", encoding="gbk", index=False)

4. Algorithmic parameter adjustment

The algorithm implemented here uses nothing more than SGD and so on, so there are also some hyperparameters that affect the final result. We can also use GridSearchCV, which is commonly used in sklearn s, to select the optimal parameters. A simple example is as follows:

# Define the grid of parameters that need to be optimized
param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005],
              'reg_all': [0.4, 0.6]}
# Cross-validation using grid search
grid_search = GridSearch(SVD, param_grid, measures=['RMSE', 'FCP'])
# Find the best parameters on the data set
data = Dataset.load_builtin('ml-100k')
data.split(n_folds=3)
grid_search.evaluate(data)
# Output tuning parameter set 
# Output the best RMSE results
print(grid_search.best_score['RMSE'])
# >>> 0.96117566386

# Output parameters corresponding to the best RMSE results
print(grid_search.best_params['RMSE'])
# >>> {'reg_all': 0.4, 'lr_all': 0.005, 'n_epochs': 10}

# Best FCP score
print(grid_search.best_score['FCP'])
# >>> 0.702279736531

# Parameters corresponding to the highest FCP score
print(grid_search.best_params['FCP'])
# >>> {'reg_all': 0.6, 'lr_all': 0.005, 'n_epochs': 10}

GridSearchCV method:

# Define the grid of parameters that need to be optimized
param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005],
              'reg_all': [0.4, 0.6]}
# Cross-validation using grid search
grid_search = GridSearchCV(SVD, param_grid, measures=['RMSE', 'FCP'], cv=3)
# Find the best parameters on the data set
data = Dataset.load_builtin('ml-100k')
# pref = cross_validate(grid_search, data, cv=3)
grid_search.fit(data)
# Output tuning parameter set
# Output the best RMSE results
print(grid_search.best_score)

5. Modeling and comparison using different recommendation system algorithms

from surprise import Dataset, print_perf
from surprise.model_selection import cross_validate
data = Dataset.load_builtin('ml-100k')
### Using Normal Predictor
from surprise import NormalPredictor
algo = NormalPredictor()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Use BaselineOnly
from surprise import BaselineOnly
algo = BaselineOnly()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Collaborative filtering using Base Edition
from surprise import KNNBasic, evaluate
algo = KNNBasic()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Using Mean Collaborative Filtering
from surprise import KNNWithMeans, evaluate
algo = KNNWithMeans()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Using collaborative filtering baseline
from surprise import KNNBaseline, evaluate
algo = KNNBaseline()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Using SVD
from surprise import SVD, evaluate
algo = SVD()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Using SVD++
from surprise import SVDpp, evaluate
algo = SVDpp()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

### Using NMF
from surprise import NMF
algo = NMF()
perf = cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=3)
print_perf(perf)

6. Recommended examples of movielens

#-*- coding:utf-8 -*-
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import os
import io
from surprise import KNNBaseline
from surprise import Dataset

import logging

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(filename)s[line:%(lineno)d] %(levelname)s %(message)s',
                    datefmt='%a， %d %b %Y %H:%M:%S')


# Training recommendation model steps: 1
def getSimModle():
    # Default loading of movielens dataset
    data = Dataset.load_builtin('ml-100k')
    trainset = data.build_full_trainset()
    #Using pearson_baseline method to calculate similarity False calculates similarity between movies based on item
    sim_options = {'name': 'pearson_baseline', 'user_based': False}
    ##Using KNNBaseline algorithm
    algo = KNNBaseline(sim_options=sim_options)
    #Training model
    algo.fit(trainset)
    return algo


# Getting id to name mapping steps: 2
def read_item_names():
    """
    //Get the mapping of movie name to movie id and movie id to movie name
    """
    file_name = (os.path.expanduser('~') +
                 '/.surprise_data/ml-100k/ml-100k/u.item')
    rid_to_name = {}
    name_to_rid = {}
    with io.open(file_name, 'r', encoding='ISO-8859-1') as f:
        for line in f:
            line = line.split('|')
            rid_to_name[line[0]] = line[1]
            name_to_rid[line[1]] = line[0]
    return rid_to_name, name_to_rid


# Recommendation steps for related movies based on the previous training model:3
def showSimilarMovies(algo, rid_to_name, name_to_rid):
    # Get raw_id of the movie Toy Story (1995)
    toy_story_raw_id = name_to_rid['Toy Story (1995)']
    logging.debug('raw_id=' + toy_story_raw_id)
    #Converting raw_id of a movie to the internal ID of the model
    toy_story_inner_id = algo.trainset.to_inner_iid(toy_story_raw_id)
    logging.debug('inner_id=' + str(toy_story_inner_id))
    #Get Recommended Movies from Model Here are 10
    toy_story_neighbors = algo.get_neighbors(toy_story_inner_id, 10)
    logging.debug('neighbors_ids=' + str(toy_story_neighbors))
    #The internal id of the model is converted to the actual movie id
    neighbors_raw_ids = [algo.trainset.to_raw_iid(inner_id) for inner_id in toy_story_neighbors]
    #Get a movie id list or a movie recommendation list
    neighbors_movies = [rid_to_name[raw_id] for raw_id in neighbors_raw_ids]
    print('The 10 nearest neighbors of Toy Story are:')
    for movie in neighbors_movies:
        print(movie)


if __name__ == '__main__':
    # Get the mapping of id to name
    rid_to_name, name_to_rid = read_item_names()

    # Training Recommendation Model
    algo = getSimModle()

    ##Display related movies
    showSimilarMovies(algo, rid_to_name, name_to_rid)

Operation results: