Early Stopping

Brief Introduction

When we train deep learning neural networks, we usually hope to obtain the best generalization performance. However, all standard deep learning neural network structures such as MLP are easy to over fit: when the error rate of the network in the training set is getting lower and lower, in fact, at a certain moment, its performance in the test set has begun to deteriorate.

PS: in this figure, due to the small disturbance of the loss of the verification set, the "U-shape" of the verification set is not very obvious.

How to slove overfitting

1. Reduce the dimension of parameter space.
2. Reduce the effective scale of each dimension.
Methods to reduce the number of parameters include green constructive learning, pruning and weight sharing. The main methods to reduce the effective scale of each parameter dimension are regularization, such as weight decay and early stopping.

Early stopping

Brief Introduction

During training, calculate the performance of the model in the verification set. When the performance of the model in the verification set begins to decline, stop training.

Specific steps

Step 1: divide the training set into training set and verification set
Step 2: the training will be conducted only on the training set, and the error of the model on the verification set will be calculated for each cycle T, for example, one cycle in every 15 epoch (mini batch) training, and the optimal model parameters under the current situation will be saved.
step3: stop training when p times of bad verification set performance is observed (P can be understood as patience value and tolerance).
Step 4: use the parameters in the last iteration result as the final parameters of the model.

Codes

The following is an example to use early stopping, using a simple three-tier GCN as an example.

Pytorch = 1.7.1 ， Python = 3.6 ，torch-geomatric = 1.7.1,  CUDA = 10.1

import random
import torch
import torch.nn.functional as F
from torch_geometric.nn import GCNConv
from torch_geometric.datasets import Planetoid
import matplotlib.pyplot as plt

# Define the network used, layer 3 GCN
class GCN_NET3(torch.nn.Module):
    '''
    three-layers GCN
    two-layers GCN has a better performance
    '''
    def __init__(self, num_features, hidden_size1, hidden_size2, classes):
        '''
        :param num_features: each node has a [1,D] feature vector
        :param hidden_size1: the size of the first hidden layer
        :param hidden_size2: the size of the second hidden layer
        :param classes: the number of the classes 
        '''
        super(GCN_NET3, self).__init__()
        self.conv1 = GCNConv(num_features, hidden_size1)
        self.relu = torch.nn.ReLU()
        self.dropout = torch.nn.Dropout(p=0.5)  # use dropout to over ove-fitting
        self.conv2 = GCNConv(hidden_size1, hidden_size2)
        self.conv3 = GCNConv(hidden_size2, classes)
        self.softmax = torch.nn.Softmax(dim=1) # each raw

    def forward(self, Graph):
        x, edge_index = Graph.x, Graph.edge_index
        out = self.conv1(x, edge_index)
        out = self.relu(out)
        out = self.dropout(out)
        out = self.conv2(out, edge_index)
        out = self.relu(out)
        out = self.dropout(out)
        out = self.conv3(out, edge_index)
        out = self.softmax(out)
        return out

def setup_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    random.seed(seed)


dataset = Planetoid(root='./', name='Cora')  # if root='./', Planetoid will use local dataset
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # use cpu or gpu
model = GCN_NET3(dataset.num_node_features, 128, 64, dataset.num_classes).to(device)
data = dataset[0].to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)  # define optimizer

# define some parameters
eval_T = 5  # evaluate period
P = 3  # patience
i = 0  # record the frequency f bad performance of validation
max_epoch = 300
setup_seed(seed=20)  # set up random seed
temp_val_loss = 99999  # initialize val loss
L = []  # store loss of training
L_val = []  # store loss of val

# training process
model.train()
for epoch in range(max_epoch):
    optimizer.zero_grad()
    out = model(data)
    loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])
    _, val_pred = model(data).max(dim=1)
    loss_val = F.cross_entropy(out[data.val_mask], data.y[data.val_mask])

    # early stopping
    if (epoch % eval_T) == 0:
        if (temp_val_loss > loss_val):
            temp_val_loss = loss_val
            torch.save(model.state_dict(), "GCN_NET3.pth")  # save th current best
            i = 0  # reset i
        else:
            i = i + 1
    if i > P:
        print("Early Stopping! Epoch : ", epoch,)
        break

    L_val.append(loss_val)
    val_corrent = val_pred[data.val_mask].eq(data.y[data.val_mask]).sum().item()
    val_acc = val_corrent / data.val_mask.sum()
    print('Epoch: {}  loss : {:.4f}  val_loss: {:.4f}  val_acc: {:.4f}'.format(epoch, loss.item(),
                                                                               loss_val.item(), val_acc.item()))
    L.append(loss.item())
    loss.backward()
    optimizer.step()

# test
model.load_state_dict(torch.load("GCN_NET3.pth"))   # load parameters of the model
model.eval()
_, pred = model(data).max(dim=1)
corrent = pred[data.test_mask].eq(data.y[data.test_mask]).sum().item()
acc = corrent / data.test_mask.sum()
print("test accuracy is {:.4f}".format(acc.item()))

# plot the curve of loss

n = [i for i in range(len(L))]
plt.plot(n, L, label='train')
plt.plot(n, L_val, label='val')
plt.legend()  # show the labels
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.show()

Result

Output result:
Early Stopping!  Epoch :  28
test accuracy is 0.8030

The test accuracy has been significantly improved. When Early stopping is not used, the test accuracy is about 76%; Now it is about 78%, up to 80.3%

Disadvantages

1. If the training ends too early, the cost function will be large, and the situation to be considered is more complex. At the same time, it is possible to obtain a local optimal solution or suboptimal solution.
2. The calculation cost will be increased during training. In practice, the evaluation cost can be reduced by parallel early stopping on a CPU or GPU independent of the main training process, or less frequent evaluation and verification sets.
3. You need to save the best training copy when you need a certain storage space.

Reference

1. Deep learning - Ian goodflow
2.GCN, GraphSAGE, and GAT are implemented using Python geometry