Python White's topic report - OpenCV matting project practice

Posted by RagingEagle on Wed, 05 Jan 2022 16:48:23 +0100

Python Xiaobai's topic report - actual combat of OpenCV matting project (4)

This series is Python Xiaobai's task "image segmentation and matting based on OpenCV".
It should be noted that this series is not a matting project tutorial of OpenCV, but a topic report on this topic. It includes a relatively complete PyQt project.
From the perspective of students' project homework report, it can still be dried out for your reference.

Welcome to pay attention "Python Xiaobai's project practice @ youcans" Original works
Python Xiaobai's topic report - actual combat of OpenCV matting project (1)
Python Xiaobai's topic report - actual combat of OpenCV matting project (2)
Python Xiaobai's topic report - actual combat of OpenCV matting project (3)
Python Xiaobai's topic report - actual combat of OpenCV matting project (4)

Chapter 3 threshold matting

The threshold processing method is intuitive and simple. It is a basic image segmentation method.

The threshold value is appropriately selected according to the overall or partial information of the image. When the pixel value is higher than the threshold value, it is set to 1 / 255, and when it is lower than the threshold value, it is set to 0. In this way, the region of interest in the image is filtered out to generate a mask, and then combined with the original image to obtain the matting image.

Because the gray level series and gray level histogram of the image to be processed are uncertain, it is necessary to select an appropriate threshold processing method for image segmentation for different images and different target prospects. In this chapter, fixed threshold, adaptive threshold and color range are used for image matting.

3.1 fixed threshold matting

Blue screen matting problem generally refers to single background color image matting with known background color, which is widely used in certificate photography and film and television production. Since the background color is known, the fixed threshold method can be used for image matting and background color replacement.

The fixed threshold matting method first converts the color image into gray image, and then threshold processes the image based on the appropriate color threshold to generate a binary mask image. The matting image is obtained by combining the mask with the original image, and the image background can be replaced by combining the mask with a new background image.

The threshold of fixed threshold processing method can be selected by manual experience or based on image histogram. For the blue screen matting problem, the fixed threshold processing method can get better results.

OpenCV provides the function cv Threshold() implements image threshold processing.

Function Description:

cv.threshold(src, thresh, maxval, type[, dst]) → ret, dst

The function threshold() converts the gray image into Binarization, that is, the image is completely composed of pixels 0 and 255, showing a visual effect of only black and white, highlighting the outline of the picture.

This method processes images through fixed threshold thresh, also known as fixed threshold processing method.

Parameter Description:

scr: grayscale image for threshold processing
thresh: threshold, value range: 0 ~ 255
maxval: fill color, value range 0 ~ 255, generally 255
Type: threshold type
- cv2.THRESH_BINARY: set 0 for pixels less than the threshold and maxval for pixels greater than the threshold;
- cv2.THRESH_BINARY_INV: set maxval for pixels less than the threshold and 0 for pixels greater than the threshold;
- cv2.THRESH_TRUNC: the pixels less than the threshold remain unchanged (keep the original value), and the pixels greater than the threshold set the threshold thresh;
- cv2.THRESH_TOZERO: set the pixel less than the threshold value to 0, and it will remain unchanged when it is greater than the threshold value (keep the original value)
- cv2.THRESH_TOZERO_INV: set the pixel greater than the threshold value to 0, and it will remain unchanged when it is less than the threshold value (keep the original value)
- cv2.THRESH_OTSU: use the OTSU algorithm to select the threshold
dst: returns a binary grayscale image
ret: returns the threshold value of binarization

The basic procedure for image matting using a fixed threshold is as follows:

# MattingThresh.py
# Copyright 2021 youcans, XUPTy
# Crated: 2021-12-10 

# 1. Read the original image
imgOri = cv2.imread("../images/lady983Green.jpg") # Read original image
width, height, channels = imgOri.shape

# 2. Extract the green channel from the original image
imgGray = cv2.cvtColor(imgOri, cv2.COLOR_BGR2GRAY) # Convert color image to gray image
imgGreen = imgOri[:,:,1] # imgGreen is the color intensity map of the green channel (note that it is not the gray conversion result of the original image)
print(imgOri.shape, imgGray.shape, imgGreen.shape)

# 3. The green channel is converted into a binary image to generate Mask mask and inverse Mask MaskInv
# If the background is not green screen but other colors, the corresponding color channel can be used for threshold processing (fixed threshold processing based on gray image is not suitable, and the performance is very different)
colorThresh = 220 # Color threshold of green screen background (pay attention to the influence of threshold)
ret, binary = cv2.threshold(imgGreen, colorThresh, 255, cv2.THRESH_BINARY) # Convert to binary image, generate mask, and black mask the matting area
binaryInv = cv2.bitwise_not(binary) # By bit non (black-and-white transpose), an inverse mask is generated, the matting area is white, the window is opened, and the area outside the matting is black

# 4. Matting and changing background with mask
# Generate matting image (foreground reserved, background black)
imgMatte = cv2.bitwise_and(imgOri, imgOri, mask=binaryInv) # Generate matting foreground, and output black in the inverse mask area other than standard matting

# Change the background color to red: modify the inverse mask (black outside the matting area)
imgReplace = imgOri.copy()
imgReplace[binaryInv==0] = [0,0,255] # The black area (0 / 0 / 0) is changed to red (BGR:0/0/255) 

plt.figure(figsize=(12,8))
plt.subplot(231), plt.imshow(cv2.cvtColor(imgOri, cv2.COLOR_BGR2RGB)), plt.title("Origin image"), plt.axis('off'
plt.subplot(232), plt.imshow(imgGray, cmap='gray'), plt.title("Gray image"), plt.axis('off')
plt.subplot(233), plt.imshow(imgGreen, cmap='gray'), plt.title("Green channel level"), plt.axis('off')
plt.subplot(234), plt.imshow(binaryInv, cmap='gray'), plt.title("inv-binary mask"), plt.axis('off')
plt.subplot(235), plt.imshow(cv2.cvtColor(imgMatte, cv2.COLOR_BGR2RGB)), plt.title("Matting Image"), plt.axis('off')
plt.subplot(236), plt.imshow(cv2.cvtColor(imgReplace, cv2.COLOR_BGR2RGB)), plt.title("BgColor changed"), plt.axis('off')
plt.tight_layout()
plt.show()

The results of matting monochrome background images using the fixed threshold method are shown in Fig. 3.1 ~ Fig. 3.3.

In the figure: inv binary mask is the generated mask image, matching image is the generated target foreground image, and BgColor changed is the composite Matting Image.

It should be noted that the result in the figure is not the result of converting the original image into a gray image for threshold processing, but the result of imaGreen processing of extracting the original image. Because the background of the original image is green, the performance of threshold processing using image green channel is significantly higher than that of gray image.

(1) Using fixed threshold method to matting monochrome background image can achieve satisfactory results;
(2) The threshold setting has a great impact on the performance of blue screen matting. Fig. 3.1 ~ Fig. 3.3 are the experimental results when the threshold is set to 220, 230 and 245 respectively.

From the comparison of the above three groups of pictures, it can be seen that the mask image is more accurate and complete when the threshold is large, and the mask image quality is reduced after the threshold is reduced.

The fixed threshold method has good performance for image matting with blue / Green / red background, but it is difficult for natural background image processing.

Figure 3.1 fixed threshold matting method (threshold = 245)