Premise: software installation and learning video explanation
When we use the neural network in openmv, there is no nn library for you to call in the new version. It needs to be in the old version. Baidu cloud link: https://pan.baidu.com/s/1bgLiLMxyqqL9X3h5dN8cZA Extraction code: behn (can you praise this alone)
The basic and use of digital recognition in this paper can be seen in this video OpenMV Lenet digital star pupil Technology
1, Theoretical knowledge - visual recognition and neural network
nn Modules are used for neural networks. nn_class objects are net.search() Return. classnn.nn_class, please call net.search() To create this object with a neural network algorithm
Neural network is inspired by neurons. The research on neurons has a long history. Biologists have known the composition and structure of neurons since 1904. A neuron usually has multiple dendrites, which are mainly used to receive incoming information; while there is only one axon, and there are many axon terminals at the end of the axon, which can transmit information to other neurons. Axon terminals are similar to other neurons The location of this connection is called "synapse" in biology. Here you can refer to the relationship between neuronal information transmission and pulse stimulation excitation in high school biology.
In 1949, psychologist Hebb proposed the Hebb learning rate, believing that the strength of synapses (i.e. connections) of human brain nerve cells can change. Therefore, computational scientists began to consider using the method of adjusting weights to make machine learning. This laid the foundation for the following learning algorithms.
In 1958, Rosenblatt, a computational scientist, proposed a neural network composed of two layers of neurons. He named it Perceptron.
In 1986, Rumelhar and Hinton proposed back propagation (BP) algorithm, which is the most famous neural network algorithm.
form
Multilayer neural network is mainly composed of three parts: input layer, hidden layers and output layers.
Each layer is composed of units. The input layer is imported from the instance feature vector of the training set, and then transmitted to the next layer through the weight of the connection node. The output of one layer is the input of the next layer. The number of hidden layers can be arbitrary. There is one input layer and one output layer. Each unit It can also be called a neural node. The above figure shows a two-layer neural network (generally the input layer does not count).
As a multilayer forward neural network, theoretically, any equation can be simulated if there are enough hidden layers and large enough training set.
Neural network can be used to solve classification problems and regression problems.
Single layer to multi-layer neural network
A single-layer neural network is composed of two-layer neural networks, which is also called Perceptron. In the three inputs, the weights of the connecting line are W1, W2 and W3 respectively. Multiply the input and the weights, and then sum them as the input of Z unit. If Z unit is function g, then z = g(a1 * w1 + a2 * w2 + a3 * w3).
In multilayer neural networks, the output is simply used as the input of the next layer, which is multiplied by the weight and then summed.
Method 1
nn_class.rect()
Returns a rectangular tuple (x, y, w, h) of nn_class bounding box for image image.draw_rectangle() etc. image method.
nn_class.x()
Returns the x coordinate (int) of the bounding box of nn_class.
You can also use the index [0] Get this value.
nn_class.y()
Returns the y coordinate (int) of the bounding box of nn_class.
You can also use the index [1] Get this value.
nn_class.w()
Returns the w coordinate (int) of the bounding box of nn_class.
You can also use the index [2] Get this value.
nn_class.h()
Returns the h coordinate (int) of the bounding box of nn_class.
You can also use the index [3] Get this value.
nn_class.index()
Returns the index (int) detected by the nn class.
You can also use the index [4] Get this value.
nn_class.value()
Returns the value detected by nn class (float).
You can also use the index [5] Get this value.
NET constructor
classnn.load(path)
Neural network from . network Binary files are loaded into memory. Neural network layers / weights / offsets / etc. are stored on the micro Python heap. A Net object that can operate on the image is returned.
Method 2
net.forward(image[, roi[, softmax=False[, dry_run=False]]])
Run the network on the image roi (automatically scale if necessary) and return a list of floating-point values of the neural network classification results.
roi Is a rectangular tuple (x, y, w, h) of the region of interest. If not specified, it is equal to the image rectangle. Operation only roi Pixels in the.
If softmax True, the sum of all outputs in the list is 1. Otherwise, any output in the list can be between 0 and 1.
take dry_run Set to True to print out the network layers being executed instead of actually executing them. This is for debugging and debugging.
net.search(image[, roi[, threshold=0.6[, min_scale=1.0[, scale_mul=0.5[, x_overlap=0[, y_overlap=0[, contrast_threshold=1[, softmax=False]]]]]]]])
Runs the network over the image roi as a sliding window. The network detector window slides across the image in multiple proportions.
Returns the result of neural network detection nn_class Object list.
roi Is a rectangular tuple (x, y, w, h) of the region of interest. If not specified, it is equal to the image rectangle. Operation only roi Pixels in the.
After running on the area in the image, the maximum detection value will exceed threshold Add the selected object to the output list.
min_scale Controls the scale of the network model. By default, the network will not scale. However, a value of 0.5 will allow you to detect objects with an image size of 50%
scale_mul Controls how many different scales are tested. The sliding window method works by multiplying the default scale by 1 scale_mul ，
And the result needs to be greater than min_scale . scale_mul The default value of 0.5 reduces each scale change by 50%. However, the value of 0.95 is only a reduction of 5%.
x_overlap Controls the percentage of overlap with the next detection area of the sliding window. A value of zero means no overlap. A value of 0.95 means 95% overlap.
y_overlap Controls the percentage of overlap with the next detection area of the sliding window. A value of zero means no overlap. A value of 0.95 means 95% overlap.
contrast_threshold Controls the threshold for skipping low contrast areas of the image. Before running nn on an area in the image, the standard deviation will be calculated on that area if the standard deviation is lower than contrast_threshold ， Skip this area.
If softmax True, the sum of all outputs in the list is 1. Otherwise, any output in the list can be between 0 and 1.
2, Code use
1. openmv visual recognition code (learning with python language)
# LetNet Example can be used directly by connecting with the computer using openmv
import sensor, image, time, os, nn
sensor.reset() # Reset and initialize the sensor.
sensor.set_contrast(3)
sensor.set_pixformat(sensor.GRAYSCALE) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
sensor.set_windowing((128, 128)) # Set 128x128 window.
sensor.skip_frames(time=100)
sensor.set_auto_gain(False)
sensor.set_auto_exposure(False)
# Load lenet network
net = nn.load('/lenet.network')
labels = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']
clock = time.clock() # Create a clock object to track the FPS.
while(True):
clock.tick() # Update the FPS clock.
img = sensor.snapshot() # Take a picture and return the image.
out = net.forward(img.copy().binary([(150, 255)], invert=True))
max_idx = out.index(max(out))
score = int(out[max_idx]*100)
if (score < 70):
score_str = "??:??%"
else:
score_str = "%s:%d%% "%(labels[max_idx], score)
img.draw_string(0, 0, score_str)
print(clock.fps()) # Note: OpenMV Cam runs about half as fast when connected
# to the IDE. The FPS should increase once disconnected.
When communicating, it is best to have a good foundation for C language, otherwise it will be difficult to prepare for the communication module.
openmv and stm32 code adjustment
# LetNet Example can be used directly by connecting with the computer using openmv
import sensor, image, time, os, nn
import pyb from uart
sensor.reset() # Reset and initialize the sensor.
sensor.set_contrast(3)
sensor.set_pixformat(sensor.GRAYSCALE) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
sensor.set_windowing((128, 128)) # Set 128x128 window.
sensor.skip_frames(time=100)
sensor.set_auto_gain(False)
sensor.set_auto_exposure(False)
uart(1,115200)#Here, you should pay attention to the serial communication of your stm32. For example, when calling UART1, you use 1
# Load lenet network
net = nn.load('/lenet.network')
labels = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9',0x5B]
clock = time.clock() # Create a clock object to track the FPS.
while(True):
clock.tick() # Update the FPS clock.
img = sensor.snapshot() # Take a picture and return the image.
out = net.forward(img.copy().binary([(150, 255)], invert=True))
max_idx = out.index(max(out))
score = int(out[max_idx]*100)
if (score < 70):
score_str = "??:??%"
else:
score_str = "%s:%d%% "%(labels[max_idx], score)
img.draw_string(0, 0, score_str)
print(clock.fps())
uart.write(labels[max_idx]) # Note: OpenMV Cam runs about half as fast when connected
# to the IDE. The FPS should increase once disconnected.
2. stm32 code - Serial Communication
(using IIC or SPI, radio wifi communication is also possible)
If you need WiFi to communicate with openmv, you'd better watch this tutorial video first. WIFI programming for OpenMV remote debugging | Xingtong Technology (singtown.com)
Additional modules or boards are required here, such as esp8266 (adriuno)
In stm32, serial communication is used to receive the data transmitted by opennmv. Here, it should be emphasized that the string transmitted by stm32 can only be decoded after corresponding character conversion.
//USART1 global interrupt service function
void USART1_IRQHandler(void)
{
char com_data;
static u8 RxBuffer1[10]={0};
static u8 RxFlag1 = 0;
u8 temp;
if( USART_GetITStatus(USART1,USART_IT_RXNE)!=RESET) //Receive interrupt
{
USART_ClearITPendingBit(USART1,USART_IT_RXNE); //Clear interrupt flag
com_data = USART_ReceiveData(USART1);
if(flag0==0)
{
if(com_data!=' ')
{
RxBuffer1[RxFlag1]=com_data;
RxFlag1++;
}
else if(com_data==' ')
{
sum=RxBuffer1[1];
printf("sum is %c\n", sum);
flag0=1;
RxFlag1=0;
}
}
else
{
if((com_data>47)&&(com_data<58))
{
RxBuffer1[RxFlag1]=com_data;
RxFlag1++;
}
else if(com_data==' ')
{
if(RxFlag1==1)
{
temp=(RxBuffer1[0]-48);
}
else if(RxFlag1==2)
{
temp=(RxBuffer1[0]-48)*10+(RxBuffer1[1]-48);
}
else if(RxFlag1==3)
{
temp=(RxBuffer1[0]-48)*100+(RxBuffer1[1]-48)*10+(RxBuffer1[2]-48);
}
Cx=temp;
printf("Cx is %d\n", Cx);
RxFlag1=0;
}
}
// else if(RxState==1)
// {
// if((com_data>47)&&(com_data<58))
// {
// RxBuffer1[RxFlag1]=com_data;
// RxFlag1++;
// }
// else if(com_data==' ')
// {
// if(RxFlag1==1)
// {
// Cw=(RxBuffer1[0]-48);
// }
// else if(RxFlag1==2)
// {
// Cw=(RxBuffer1[0]-48)*10+(RxBuffer1[1]-48);
// }
// else if(RxFlag1==3)
// {
// Cw=(RxBuffer1[0]-48)*100+(RxBuffer1[1]-48)*10+(RxBuffer1[2]-48);
// }
// RxFlag1=0;
// printf("Cw is %d\n", Cw);
// RxState=0;
// }
// }
}
}
Here, I will directly give the code of the interrupt function first, and other configurations are the same as those of the general stm32 serial communication. Here, I can explain why the string passed from openmv is processed by openmv, because although openmv has its own stm32 chip, it is encapsulated in python for better simplification in the future, but it is not specified in python All outgoing data is the default transmission string. The rest is to check whether there is data from the serial port assistant