# Basic structure of TCN

The time-domain convolution data proposed by temporal. Network in 2018 can be seen in detail paper.

## 1. Causal Convolution

Causal convolution is shown in the figure above. For the value of t-Time of the upper layer, it only depends on the value of t-Time of the lower layer and its previous value. The difference from the traditional convolution neural network is that causal convolution can not see the future data. It is a one-way structure, not two-way. In other words, there must be antecedents before there can be consequences. It is a strict time constraint model, so it is called causal convolution.

## Dilated Convolution

Dilated Convolution is also called void convolution

Pure causal convolution still has the problem of traditional convolution neural network, that is, the length of modeling time is limited by the size of convolution kernel. If you want to get longer dependencies, you need to stack many layers linearly. In order to solve this problem, researchers proposed expansion convolution, as shown in the figure below.

Unlike the traditional convolution, the expansion convolution allows the input to be sampled at intervals during the convolution, and the sampling rate is determined by d d d) control. Bottom d = 1 d = 1 d=1 means that each point is sampled in the process of input, and the middle layer d = 2 d = 2 d=2 means that every two points are sampled as input during the input process. Generally speaking, the higher the level, the greater the value of d. Therefore, the expansion convolution makes the size of the effective window increase exponentially with the number of layers. By this method, the convolution network can use fewer layers and obtain a large receptive field.

## Residual Connections

Residual connection is proved to be an effective method to train deep networks, which allows the network to transmit information in a cross layer manner.

In this paper, a residual block is constructed to replace the convolution layer. As shown in the figure above, a residual block contains two layers of convolution and nonlinear mapping, and WeightNorm and Dropout are added to each layer to regularize the network.

# TCN summary

## advantage

(1) Parallelism. When a sentence is given, TCN can process sentences in parallel without sequential processing like RNN.

(2) Flexible receptive field. The size of the receptive field of TCN is determined by the number of layers, the size of convolution kernel and expansion coefficient. It can be customized flexibly according to different tasks and different characteristics.

(3) Gradient stability. RNN often has the problems of gradient disappearance and gradient explosion, which is mainly caused by sharing parameters in different time periods. Like the traditional convolutional neural network, TCN does not have the problems of gradient disappearance and gradient explosion.

(4) Lower memory. When using RNN, it needs to save the information of each step, which will occupy a lot of memory. The convolution kernel of TCN is shared in one layer, and the memory usage is lower.

## shortcoming

(1) TCN may not be so adaptable in transfer learning. This is because the amount of historical information required for model prediction may be different in different fields. Therefore, when a model is migrated from a problem requiring less memory information to a problem requiring longer memory, the performance of TCN may be very poor because its receptive field is not large enough.

(2) The TCN described in this paper is still a unidirectional structure. Pure unidirectional structures are still very useful in tasks such as speech recognition and speech synthesis. However, most texts use a two-way structure. TCN can be easily extended to bidirectional structure, just use the traditional convolution structure instead of causal convolution.

(3) After all, TCN is a variant of convolutional neural network. Although the use of extended convolution can expand the receptive field, it is still limited. Compared with Transformer, the feature that any length of relevant information can be obtained is still very poor. The application of TCN in text remains to be tested.

# TCN application

## MINST handwritten numeral classification

Multiple features correspond to one label, i.e. (xi1,xi2,xi3,... xin) - yi

Local environment:

Python 3.6 IDE:Pycharm

Library version:

keras 2.2.0 numpy 1.16.2 tensorflow 1.9.0

1. Download data set

2. Create TCN Py, enter the following code

Code reference: Keras-based TCN

# TCN for minst data from tensorflow.examples.tutorials.mnist import input_data from keras.models import Model from keras.layers import add, Input, Conv1D, Activation, Flatten, Dense # Load data load data def read_data(path): mnist = input_data.read_data_sets(path, one_hot=True) train_x, train_y = mnist.train.images.reshape(-1, 28, 28), mnist.train.labels, valid_x, valid_y = mnist.validation.images.reshape(-1, 28, 28), mnist.validation.labels, test_x, test_y = mnist.test.images.reshape(-1, 28, 28), mnist.test.labels return train_x, train_y, valid_x, valid_y, test_x, test_y # Residual block def ResBlock(x, filters, kernel_size, dilation_rate): r = Conv1D(filters, kernel_size, padding='same', dilation_rate=dilation_rate, activation='relu')( x) # first convolution r = Conv1D(filters, kernel_size, padding='same', dilation_rate=dilation_rate)(r) # Second convolution if x.shape[-1] == filters: shortcut = x else: shortcut = Conv1D(filters, kernel_size, padding='same')(x) # shortcut (shortcut) o = add([r, shortcut]) # Activation function o = Activation('relu')(o) return o # Sequence Model def TCN(train_x, train_y, valid_x, valid_y, test_x, test_y, classes, epoch): inputs = Input(shape=(28, 28)) x = ResBlock(inputs, filters=32, kernel_size=3, dilation_rate=1) x = ResBlock(x, filters=32, kernel_size=3, dilation_rate=2) x = ResBlock(x, filters=16, kernel_size=3, dilation_rate=4) x = Flatten()(x) x = Dense(classes, activation='softmax')(x) model = Model(input=inputs, output=x) # View network structure model.summary() # Compile model model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) # Training model model.fit(train_x, train_y, batch_size=500, nb_epoch=epoch, verbose=2, validation_data=(valid_x, valid_y)) # Assessment model pre = model.evaluate(test_x, test_y, batch_size=500, verbose=2) print('test_loss:', pre[0], '- test_acc:', pre[1]) # There are 10 MINST numbers from 0-9, i.e. 10 categories classes = 10 epoch = 30 train_x, train_y, valid_x, valid_y, test_x, test_y = read_data('MNIST_data') #print(train_x, train_y) TCN(train_x, train_y, valid_x, valid_y, test_x, test_y, classes, epoch)

3. Results

test_loss: 0.05342669463425409 - test_acc: 0.987100002169609

## Multiple labels

Multiple features correspond to multiple labels, such as (xi1,xi2,xi3,... xin) - (yi1, yi2)

Just modify the above code, reconstruct the training and test data, and set the corresponding input and output dimensions, parameters and other information.

Local environment:

Python 3.6 IDE:Pycharm

Library version:

keras 2.2.0 numpy 1.16.2 pandas 0.24.1 sklearn 0.20.1 tensorflow 1.9.0

Specific code:

# TCN for indoor location import math from tensorflow.examples.tutorials.mnist import input_data from keras.models import Model from keras.layers import add, Input, Conv1D, Activation, Flatten, Dense import numpy as np import pandas from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import mean_squared_error # Create timing data def create_dataset(dataset, look_back=1): dataX, dataY = [], [] for i in range(len(dataset)-look_back-1): a = dataset[i:(i+look_back), :] dataX.append(a) dataY.append(dataset[i + look_back, -2:]) return np.array(dataX), np.array(dataY) # Residual block def ResBlock(x, filters, kernel_size, dilation_rate): r = Conv1D(filters, kernel_size, padding='same', dilation_rate=dilation_rate, activation='relu')( x) # first convolution r = Conv1D(filters, kernel_size, padding='same', dilation_rate=dilation_rate)(r) # Second convolution if x.shape[-1] == filters: shortcut = x else: shortcut = Conv1D(filters, kernel_size, padding='same')(x) # shortcut (shortcut) o = add([r, shortcut]) o = Activation('relu')(o) # Activation function return o # Sequence Model def TCN(train_x, train_y, test_x, test_y, look_back, n_features, n_output, epoch): inputs = Input(shape=(look_back, n_features)) x = ResBlock(inputs, filters=32, kernel_size=3, dilation_rate=1) x = ResBlock(x, filters=32, kernel_size=3, dilation_rate=2) x = ResBlock(x, filters=16, kernel_size=3, dilation_rate=4) x = Flatten()(x) x = Dense(n_output, activation='softmax')(x) model = Model(input=inputs, output=x) # View network structure model.summary() # Compile model model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) # Training model model.fit(train_x, train_y, batch_size=500, nb_epoch=epoch, verbose=2) # Assessment model pre = model.evaluate(test_x, test_y, batch_size=500, verbose=2) # print(pre) print('test_loss:', pre[0], '- test_acc:', pre[1]) # Common parameters np.random.seed(7) features = 24 output = 2 EPOCH = 30 look_back = 5 trainPath = '../data/train.csv' testPath = '../data/test.csv' trainData = pandas.read_csv(trainPath, engine='python') testData = pandas.read_csv(testPath, engine='python') # features = 1 dataset = trainData.values dataset = dataset.astype('float32') datatestset = testData.values datatestset = datatestset.astype('float32') # print(dataset) # normalize the dataset scaler = MinMaxScaler(feature_range=(0, 1)) dataset = scaler.fit_transform(dataset) datatestset = scaler.fit_transform(datatestset) trainX, trainY = create_dataset(dataset, look_back) testX, testY = create_dataset(datatestset, look_back) # print(trainX) print(len(trainX), len(testX)) print(testX.shape) # reshape input to be [samples, time steps, features] trainX = np.reshape(trainX, (trainX.shape[0], trainX.shape[1], features)) testX = np.reshape(testX, (testX.shape[0], testX.shape[1], features)) # train_x, train_y, valid_x, valid_y, test_x, test_y = read_data('MNIST_data') print(trainX, trainY) TCN(trainX, trainY, testX, testY, look_back, features, output, EPOCH)

To be continued

# reference material

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

TCN-Time Convolutional Network