Rimeng Society
2.4 Cases: Realizing Linear Regression
Learning Objectives
- target
- Applying op's name parameter to modify op's name
- Apply variable_scope implementation diagram program scope addition
- Tracking display of tensor values using scalar or histogram
- Apply merge_all implements the merging of tensor values
- Apply add_Summy implements writing tensor values to files
- applicationTf.train.saverImplement model saving and loading for TensorFlow
- applicationTf.app.flagsImplement command line parameter addition and use
- Apply reduce_mean, square to realize mean square error calculation
- applicationTf.train.GradientDescentOptimizerImplement gradient descent optimizer creation
- Optimize loss with minimize function
- Know gradient explosions and common solutions
- application
- Implementing linear regression models
- Content Preview
- Review of 2.6.1 Linear Regression Principle
- 2.6.2 Cases: Training for Linear Regression
- 2.6.3 Additional features
- 1 Increase variable display
- 2 Increase Namespace
- 3 Save and Load Model
- 4 Use of command line parameters
Review of 2.4.1 Linear Regression Principle
Regression model based on data, w1x1+w2x2+.... +b = y, establishes the error between the true value and the predicted value, and uses gradient descent optimization to obtain the weight and offset corresponding to the minimum loss.Finally, the weights and offset parameters of the model are determined.Finally, these parameters can be used to make predictions.
2.4.2 Cases: Training for Linear Regression
1 Case Determination
- Assume that 100 points are randomly assigned with only one feature
- The distribution of the data itself is y = 0.8 * x + 0.7
The rule of data distribution is determined here to compare the training parameters with the actual parameters (i.e. 0.8 and 0.7) and whether they are accurate or not.
2-step analysis
- 1 Prepare the dataset: y = 0.8x + 0.7 100 samples
- 2 Establish a linear model
- Random Initialization W1 and b1
- y = W.X + b, objective: find the weight W and offset B
- 3 Determine loss function (error between predicted and true values) - mean square error
- 4 Gradient Down Optimized Loss: Need to specify learning rate (hyperparameter)
4 Implement full functionality
Related API s
operation
- Matrix operations
- tf.matmul(x, w)
- square
- tf.square(error)
- mean value
- tf.reduce_mean(error)
error = tf.reduce_mean(tf.square(y_predict - y_true))
Gradient descent optimization
- tf.train.GradientDescentOptimizer(learning_rate)
- Gradient descent optimization
- learning_rate: learning rate, typically a small value between 0 and 1
- method:
- minimize(loss)
- return:gradient descent op
# Tensorflow provides a variety of optimization methods, no need to implement gradient descent operation principle formulas by itself, custom use of optimization types optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(error)
Complete Code Process Analysis:
import tensorflow as tf
import os
def linear_regression():
"""
//Self-realizing linear regression
:return: None
"""
# 1) Prepare the dataset: y = 0.8x + 0.7 100 samples
# Eigenvalue X, Target Value y_true
X = tf.random_normal(shape=(100, 1), mean=2, stddev=2)
# y_true [100, 1]
# Matrix operation X(100,1)*(1,1)= y_true(100, 1)
y_true = tf.matmul(X, [[0.8]]) + 0.7
# 2) Establish a linear model:
# y = W.X + b, objective: find the weight W and offset B
# 3) Initialize W1 and b1 randomly
weights = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)))
bias = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)))
y_predict = tf.matmul(X, weights) + bias
# 4) Determine the loss function (the error between the predicted value and the true value) - the mean square error
error = tf.reduce_mean(tf.square(y_predict - y_true))
# 5) Optimized loss of gradient descent: Need to specify learning rate (hyperparameter)
# W2 = W1 - Learning rate* (Direction)
# b2 = b1 - learning rate* (direction)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(error)
# initialize variable
init = tf.global_variables_initializer()
# Open a session for training
with tf.Session() as sess:
# Run Initialization Variable Op
sess.run(init)
print("Randomly initialized weights are%f， Offset to%f" % (weights.eval(), bias.eval()))
# Training model
for i in range(100):
sess.run(optimizer)
print("No.%d The step error is%f，Weight is%f， Offset to%f" % (i, error.eval(), weights.eval(), bias.eval()))
return None
Observation of trainable settings for 6 variables
Parameter role of trainable, specifying whether or not to train
weight = tf.Variable(tf.random_normal([1, 1], mean=0.0, stddev=1.0), name="weights", trainable=False)
2.4.3 Additional features
- Increase Namespace
- Command Line Parameter Settings
2 Increase Namespace
Is the code structure clearer, Tensorboard diagram structure clearer
with tf.variable_scope("lr_model"):
def linear_regression():
# 1) Prepare the dataset: y = 0.8x + 0.7 100 samples
# Eigenvalue X, Target Value y_true
with tf.variable_scope("original_data"):
X = tf.random_normal(shape=(100, 1), mean=2, stddev=2, name="original_data_x")
# y_true [100, 1]
# Matrix operation X(100,1)*(1,1)= y_true(100, 1)
y_true = tf.matmul(X, [[0.8]], name="original_matmul") + 0.7
# 2) Establish a linear model:
# y = W.X + b, objective: find the weight W and offset B
# 3) Initialize W1 and b1 randomly
with tf.variable_scope("linear_model"):
weights = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)), name="weights")
bias = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)), name="bias")
y_predict = tf.matmul(X, weights, name="model_matmul") + bias
# 4) Determine the loss function (the error between the predicted value and the true value) - the mean square error
with tf.variable_scope("loss"):
error = tf.reduce_mean(tf.square(y_predict - y_true), name="error_op")
# 5) Optimized loss of gradient descent: Need to specify learning rate (hyperparameter)
# W2 = W1 - Learning rate* (Direction)
# b2 = b1 - learning rate* (direction)
with tf.variable_scope("gd_optimizer"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01, name="optimizer").minimize(error)
# 2) Collect variables
tf.summary.scalar("error", error)
tf.summary.histogram("weights", weights)
tf.summary.histogram("bias", bias)
# 3) Merge variables
merge = tf.summary.merge_all()
# initialize variable
init = tf.global_variables_initializer()
# Open a session for training
with tf.Session() as sess:
# Run Initialization Variable Op
sess.run(init)
print("Randomly initialized weights are%f， Offset to%f" % (weights.eval(), bias.eval()))
# 1) Create event file
file_writer = tf.summary.FileWriter(logdir="./summary", graph=sess.graph)
# Training model
for i in range(100):
sess.run(optimizer)
print("No.%d The step error is%f，Weight is%f， Offset to%f" % (i, error.eval(), weights.eval(), bias.eval()))
# 4) Run the merge variable op
summary = sess.run(merge)
file_writer.add_summary(summary, i)
return None
3 Save and Load Model
-
tf.train.Saver(var_list=None,max_to_keep=5)
- Save and load the model (Save file format: checkpoint file)
- var_list: Specifies the variables to be saved and restored.It can be passed as a dict or as a list.
- max_to_keep: Indicates the maximum number of recent checkpoint files to keep.When you create a new file, older files are deleted.If none or 0, all checkpoint files are preserved.The default is 5 (that is, keep the latest five checkpoint files.)
Use
For example: Specify Catalog + Model Name saver.save(sess, '/tmp/ckpt/test/myregression.ckpt') saver.restore(sess, '/tmp/ckpt/test/myregression.ckpt')
To determine if a model exists, specify a directory directly
checkpoint = tf.train.latest_checkpoint("./tmp/model/")
saver.restore(sess, checkpoint)
4 Use of command line parameters
- 1,tf.app.flags., there is a FLAGS flag in flags that can be called into our programs
Flag_as specifically defined aboveName
- 2. PassTf.app.run() Start main(argv) function
# Define some common command line parameters
# Training steps
tf.app.flags.DEFINE_integer("max_step", 0, "Number of steps in training model")
# Define Path to Model
tf.app.flags.DEFINE_string("model_dir", " ", "Path to model save+Model name")
# Define Get Command Line Parameters
FLAGS = tf.app.flags.FLAGS
# Open Training
# Number of training steps (depending on model size)
for i in range(FLAGS.max_step):
sess.run(train_op)
Complete Code
import tensorflow as tf
import os
tf.app.flags.DEFINE_string("model_path", "./linear_regression/", "Path and file name to save the model")
FLAGS = tf.app.flags.FLAGS
def linear_regression():
# 1) Prepare the dataset: y = 0.8x + 0.7 100 samples
# Eigenvalue X, Target Value y_true
with tf.variable_scope("original_data"):
X = tf.random_normal(shape=(100, 1), mean=2, stddev=2, name="original_data_x")
# y_true [100, 1]
# Matrix operation X(100,1)*(1,1)= y_true(100, 1)
y_true = tf.matmul(X, [[0.8]], name="original_matmul") + 0.7
# 2) Establish a linear model:
# y = W.X + b, objective: find the weight W and offset B
# 3) Initialize W1 and b1 randomly
with tf.variable_scope("linear_model"):
weights = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)), name="weights")
bias = tf.Variable(initial_value=tf.random_normal(shape=(1, 1)), name="bias")
y_predict = tf.matmul(X, weights, name="model_matmul") + bias
# 4) Determine the loss function (the error between the predicted value and the true value) - the mean square error
with tf.variable_scope("loss"):
error = tf.reduce_mean(tf.square(y_predict - y_true), name="error_op")
# 5) Optimized loss of gradient descent: Need to specify learning rate (hyperparameter)
# W2 = W1 - Learning rate* (Direction)
# b2 = b1 - learning rate* (direction)
with tf.variable_scope("gd_optimizer"):
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01, name="optimizer").minimize(error)
# 2) Collect variables
tf.summary.scalar("error", error)
tf.summary.histogram("weights", weights)
tf.summary.histogram("bias", bias)
# 3) Merge variables
merge = tf.summary.merge_all()
# initialize variable
init = tf.global_variables_initializer()
# Open a session for training
with tf.Session() as sess:
# Run Initialization Variable Op
sess.run(init)
# Untrained weights and offsets
print("Randomly initialized weights are%f， Offset to%f" % (weights.eval(), bias.eval()))
# Load the model when a checkpoint file exists
# 1) Create event file
file_writer = tf.summary.FileWriter(logdir="./summary", graph=sess.graph)
# Training model
for i in range(100):
sess.run(optimizer)
print("No.%d The step error is%f，Weight is%f， Offset to%f" % (i, error.eval(), weights.eval(), bias.eval()))
# 4) Run the merge variable op
summary = sess.run(merge)
file_writer.add_summary(summary, i)
return None
def main(argv):
print("This is main function")
print(argv)
print(FLAGS.model_path)
linear_regression()
if __name__ == "__main__":
tf.app.run()
Job: Change process-oriented to object-oriented
Reference Code
# Self-implementation of a linear regression case using tensorflow
# Define some common command line parameters
# Training steps
tf.app.flags.DEFINE_integer("max_step", 0, "Number of steps in training model")
# Define Path to Model
tf.app.flags.DEFINE_string("model_dir", " ", "Path to model save+Model name")
FLAGS = tf.app.flags.FLAGS
class MyLinearRegression(object):
"""
//Self-realizing linear regression
"""
def __init__(self):
pass
def inputs(self):
"""
//Getting eigenvalue target data
:return:
"""
x_data = tf.random_normal([100, 1], mean=1.0, stddev=1.0, name="x_data")
y_true = tf.matmul(x_data, [[0.7]]) + 0.8
return x_data, y_true
def inference(self, feature):
"""
//Modeling from input data
:param feature:
:param label:
:return:
"""
with tf.variable_scope("linea_model"):
# 2. Establish a regression model to analyze the number of features of others'data - > the number of weights, offset b
# Since the gradient descent algorithm is optimized, random parameters, weights, and offsets are initially given
# Optimized parameters must be defined using variable op
# Variable Initialization Weights and Offsets
# weight 2-dimensional [1, 1] bias [1]
# There is a trainable parameter in the variable op that determines whether to train or not
self.weight = tf.Variable(tf.random_normal([1, 1], mean=0.0, stddev=1.0),
name="weights")
self.bias = tf.Variable(0.0, name='biases')
# Establish regression formulas to get prediction results
y_predict = tf.matmul(feature, self.weight) + self.bias
return y_predict
def loss(self, y_true, y_predict):
"""
//Target value and true value to calculate loss
:return: loss
"""
# 3. Find out the loss between our model and the real data
# Mean Square Error Formula
loss = tf.reduce_mean(tf.square(y_true - y_predict))
return loss
def merge_summary(self, loss):
# 1. Value of collection tensor
tf.summary.scalar("losses", loss)
tf.summary.histogram("w", self.weight)
tf.summary.histogram('b', self.bias)
# 2. Merge Variables
merged = tf.summary.merge_all()
return merged
def sgd_op(self, loss):
"""
//Get training OP
:return:
"""
# 4. Optimize using gradient descent optimizer
# Fill-in learning rate: 0-1 learning rate is very small,
# The learning rate determines how many steps you lose to reach
# Minimize loss
train_op = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
return train_op
def train(self):
"""
//Training model
:param loss:
:return:
"""
g = tf.get_default_graph()
with g.as_default():
x_data, y_true = self.inputs()
y_predict = self.inference(x_data)
loss = self.loss(y_true, y_predict)
train_op = self.sgd_op(loss)
# Collect observed result values
merged = self.merge_summary(loss)
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
# Without training, model parameter values
print("Initialized weight:%f, Offset:%f" % (self.weight.eval(), self.bias.eval()))
# Open Training
# Number of training steps (depending on model size)
for i in range(FLAGS.max_step):
sess.run(train_op)
# Generate event file, view diagram structure
file_writer = tf.summary.FileWriter("./tmp/summary/", graph=sess.graph)
print("Training session%d Loss after step:%f, Weight:%f, Offset:%f" % (
i,
loss.eval(),
self.weight.eval(),
self.bias.eval()))
# Run the results of collecting variables
summary = sess.run(merged)
# Add to File
file_writer.add_summary(summary, i)
if __name__ == '__main__':
lr = MyLinearRegression()
lr.train()