Practice the fashion MNIST dataset

Like the theoretical knowledge of Chpater3 in the book (hands on machine learning with scikit learning), it can be read together, but the MNIST data set of Chapter 3 in the book cannot be imported at all

import torch
import torchvision
from torch.utils import data
from torchvision import transforms
from d2l import  torch as d2l
from sklearn.datasets import fetch_openml
import matplotlib as mpl
import matplotlib.pyplot as plt
import os

d2l.use_svg_display()

Because the import of. D2L did not start use_ svg_ Display(), which makes the following Timer function unusable and reports an error

1. Download and read FASHION-MNIST data into memory through the built-in function in the framework

The image data is transformed from pil type to 32-bit float by to tensor, and then divided by 255, so that the values of all image pixels are between 0 and 1

trans = transforms.ToTensor() #Turn the picture into a tent of pytorch (the simplest preprocessing)
mnist_train = torchvision.datasets.FashionMNIST(root= '../data',train =True,transform=trans, download=True ) #Get the train data from the dataset and get the tensor of pytorch
mnist_test = torchvision.datasets.FashionMNIST(root='../data',train=False,transform=trans,download=True) #The test data is a data set used to determine whether the model is good or bad. It does not participate in training. train=False

print('Test data len:',len(mnist_test))
print('Train data len:',len(mnist_train))

Printed results:

Test data len: 10000
Train data len: 60000
 Fashion MNIST is composed of 10 categories of images, and each category is composed of 6000 images in the training dataset and 1000 images in the test dataset.
Therefore, the training set and the test set contain 60000 and 10000 images, respectively. The test data set will not be used for training, but only for evaluating the performance of the model. d

Print the first picture:

print(mnist_train[0][0].shape) 
#torch.Size([1, 28, 28]), black-and-white picture, rgb channel=1, and the length and width of the picture are 28
#Print the first picture, even if minist_train[0][0]

2. Define the function of visual dataset (this step is the same as the last manual dataset)

def get_fashion_mnist_labels(labels): #Returns the text label of the FASHION-MNIST dataset
    text_labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
                   'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']
    #There are 10 categories in the dataset
    return [text_labels[int(i)]for i in labels]

3. We can now create a function to visualize these samples.

def show_images(imgs,num_rows,num_cols,titles=None,scale=1.5):
    #Drawing with matplotlip
    figsize = (num_cols * scale,num_rows * scale)
    _,axes = d2l.plt.subplots(num_rows,num_cols,figsize=figsize)
    axes = axes.flatten()
    for i, (ax,img) in enumerate(zip(axes,imgs)):
        if torch.is_tensor(img):
            ax.imshow(img.numpy())
            #Picture tensor
        else:
            #PIL picture
            ax.imshow(img)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)
        if titles:
            ax.set_title(titles[i])
    return axes

#The following are the images of the first few samples in the training data set and their corresponding labels.
X,y = next(iter(data.DataLoader(mnist_train,batch_size = 18)))

plt_img = show_images(X.reshape(18, 28, 28), 2, 9, titles=get_fashion_mnist_labels(y))
plt.show()

After constructing the data set of pytorch, put it into a dataloader and specify a batch size, and you will get a batch data with a fixed size
 After constructing an interval with iter function, use next to obtain the first small batch, which is recorded as X and y
 Although the data has changed, it is essentially the same as the previous manual data exercise

Because the channel is not used, the X Reshape is changed into a number of examples. The width and height remain the same, or 28
 When drawing, draw 2 lines, 9 pictures in each line, and the label of each picture is the previous label
 The previous picture cannot be displayed because there is no PLT In the line of show (), the matplt drawing function will not be used

https://zhuanlan.zhihu.com/p/109245779

#Read a small batch of data, the size is batch size
batch_size = 256


def get_dataloader_workers():
    #A small function is defined. The value returned by the small function is 4. 4 is the number of processes reading data. Choose the size yourself
    return 4


timer = d2l.Timer()
#Function used to test speed
for X,y in train_iter_py: #Access all batch es
    continue
print(f'{timer.stop():2f} sec')

Practice using a different number of processes to see how the time changes
batch_size =256, return = 5, 2.793629 sec
batch_size =256, return = 55，12.512462 sec
 *Our suggested max number of worker in current system is 12
batch_size =256, return = 1，5.642448 sec