(Reprint) Android camera development: efficient real-time processing of preview frame data

Posted by domainbanshee on Thu, 14 Nov 2019 13:58:35 +0100

Summary

In this article, we will not introduce the new functions of Camera APP, but how to process the Camera preview frame data. Presumably most people don't need to deal with preview frames, because cameras only need to be used for taking photos and recording. In fact, this article doesn't have much connection with general Camera development, but it is still operating Camera class, so it is still classified as Camera development. To process preview frames is simply to process the data of each frame when the Camera previews. Generally speaking, if the sampling rate of the Camera is 30 FPS, there will be 30 frames of data to be processed in one second. What is the frame data? If you are rushing to deal with frame data, you must have known the answer for a long time. In fact, it is an array of byte type, which contains frame data in YUV format. This paper only introduces several efficient methods to deal with preview frame data, but not the specific use, because it is a long story to use for face recognition, image beautification and so on.

This article is in Android camera development (2): add preferences to the camera Based on the introduction. The processing of preview frame data usually involves a lot of calculation, which leads to the low efficiency of processing because of too much frame data, and the resulting preview picture stuck and other problems. This paper mainly introduces how to separate threads and optimize the screen display, and how to improve the efficiency of frame data processing by using HandlerThread, Queue, ThreadPool and AsyncTask.

Get ready

For the sake of simplicity, we start to get the preview frame and process it when the camera starts to preview. In order to analyze the process more clearly, we add "start" and "stop" buttons under the "Settings" button in the UI to control the start and stop of the camera preview.

Modify UI

Modify activity main.xml to

Java

<Button
    android:id="@+id/button_settings"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:text="Set up" />

Replace with

Java

<LinearLayout
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_gravity="right"
    android:orientation="vertical">

    <Button
        android:id="@+id/button_settings"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Set up" />

    <Button
        android:id="@+id/button_start_preview"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="start" />

    <Button
        android:id="@+id/button_stop_preview"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Stop it" />
</LinearLayout>

In this way, two buttons "start" and "stop" are added.

Binding events

Modify mainActivity, and transfer the code of initializing camera preview in the original onCreate() to the new method startPreview()

Java

public void startPreview() {
    final CameraPreview mPreview = new CameraPreview(this);
    FrameLayout preview = (FrameLayout) findViewById(R.id.camera_preview);
    preview.addView(mPreview);

    SettingsFragment.passCamera(mPreview.getCameraInstance());
    PreferenceManager.setDefaultValues(this, R.xml.preferences, false);
    SettingsFragment.setDefault(PreferenceManager.getDefaultSharedPreferences(this));
    SettingsFragment.init(PreferenceManager.getDefaultSharedPreferences(this));

    Button buttonSettings = (Button) findViewById(R.id.button_settings);
    buttonSettings.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            getFragmentManager().beginTransaction().replace(R.id.camera_preview, new SettingsFragment()).addToBackStack(null).commit();
        }
    });
}

At the same time, a stopPreview() method is added to stop the camera preview

Java

public void stopPreview() {
    FrameLayout preview = (FrameLayout) findViewById(R.id.camera_preview);
    preview.removeAllViews();
}

stopPreview() gets the FrameLayout where the camera preview is located, and then removes the camera preview through removeAllViews(). At this time, the relevant end method in the CameraPreview class will be triggered to close the camera preview.

Now onCreate() is very simple, just bind two buttons to corresponding methods

Java

protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    Button buttonStartPreview = (Button) findViewById(R.id.button_start_preview);
    buttonStartPreview.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            startPreview();
        }
    });
    Button buttonStopPreview = (Button) findViewById(R.id.button_stop_preview);
    buttonStopPreview.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            stopPreview();
        }
    });
}

Try running

Now running APP will not start camera preview immediately. Click the "start" button to display the camera preview screen. Click "stop" to make the screen disappear and preview stop.

Basic frame data acquisition and processing

Here we first implement the most basic and commonly used method of frame data acquisition and processing; then we look at the methods to improve the performance.

Basics

The interface to get frame data is Camera.PreviewCallback. To achieve the onpreviewframe (byte [] data, Camera) method under this interface, you can get the data of each frame. So what we need to do now is to add the Camera.PreviewCallback interface declaration to the CameraPreview class, then implement the onPreviewFrame() method in CameraPreview, and finally bind the interface to Camera. In this way, for each preview frame generated during Camera preview, the onPreviewFrame() method will be called to process the preview frame data.

In CameraPreview, modify

Java

public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback

Java

public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback, Camera.PreviewCallback

Add Camera.PreviewCallback interface declaration.

Add the implementation of onPreviewFrame()

Java

public void onPreviewFrame(byte[] data, Camera camera) {
    Log.i(TAG, "processing frame");
    try {
        Thread.sleep(500);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

Instead of processing frame data, we pause 0.5 seconds to simulate processing frame data.

Add the sentence getCameraInstance() in surfaceCreated()

Java

mCamera.setPreviewCallback(this);

Bind this interface to mCamera so that onPreviewFrame() is called whenever a preview frame is generated.

Try running

Now run the APP and click "start". Generally, no obvious difference can be observed on the screen, but there are actually two potential problems. First, if you click "Settings" at this time, you will find that the setting interface does not appear immediately, but will appear a few seconds later; and then click the back button, the setting interface will disappear in a few seconds. Second, you can see the output "processing frame" in logcat, which is about 0.5 seconds. Because the thread sleep is set to 0.5 seconds, only 2 frames of 30 frames per second are processed, and the remaining 28 frames are discarded (there is no very intuitive way to show that the remaining 28 frames are discarded, but the fact is that, not strictly speaking, when the new frame data arrives If onPreviewFrame() is executing and has not returned, the frame data will be discarded.

Separate from UI thread

problem analysis

Now let's solve the first problem. The first reason is simple, which is often encountered in Android Development: the UI thread is occupied, resulting in the UI operation stuck. In this case, onPreviewFrame() will block the thread, and the blocked thread is the UI thread.

On which thread does onPreviewFrame() execute? There are descriptions in the official documents:

Called as preview frames are displayed. This callback is invoked on the event thread open(int) was called from.

This means that onPreviewFrame() runs on the same thread as Camera.open(). At present, Camera.open() is executed in UI thread (because no new process has been created). The corresponding solution is simple: let Camera.open() execute in non UI thread.

Solution

Here we use HandlerThread to implement. HandlerThread will create a new thread with its own loop, so that the specified statement can be executed in the new thread through Handler.post(). Although it's easy to say, there are still some details to deal with.

Start with HandlerThread and add it to CameraPreview

Java

private class CameraHandlerThread extends HandlerThread {
    Handler mHandler;

    public CameraHandlerThread(String name) {
        super(name);
        start();
        mHandler = new Handler(getLooper());
    }

    synchronized void notifyCameraOpened() {
        notify();
    }

    void openCamera() {
        mHandler.post(new Runnable() {
            @Override
            public void run() {
                openCameraOriginal();
                notifyCameraOpened();
            }
        });
        try {
            wait();
        } catch (InterruptedException e) {
            Log.w(TAG, "wait was interrupted");
        }
    }
}

CameraHandlerThread inherits from HandlerThread. Start the Thread in the constructor and create a handler. The effect of openCamera() is to execute mCamera = Camera.open(); in this Thread, so it is executed in Runnable() through handler.post(), and the statement to be executed is encapsulated in openCameraOriginal(). The use of notify wait is for security, because post() execution will return immediately, while Runnable() will execute asynchronously, which may still be null when using mmcamera immediately after post(); therefore, add the notify wait control here, and openCamera() will return only after confirming that the camera is turned on.

Next, openCameraOriginal() is added to CameraPreview

Java

private void openCameraOriginal() {
    try {
        mCamera = Camera.open();
    } catch (Exception e) {
        Log.d(TAG, "camera is not available");
    }
}

This does not need to be explained, but encapsulation becomes a method.

Finally, change getCameraInstance() to

Java

public Camera getCameraInstance() {
    if (mCamera == null) {
        CameraHandlerThread mThread = new CameraHandlerThread("camera thread");
        synchronized (mThread) {
            mThread.openCamera();
        }
    }
    return mCamera;
}

It's also easy to understand. It's for camera handler thread to handle.

Try running

Now running the APP, you will find that the first problem has been solved.

Processing frame data

Next, to solve the second problem, how to ensure that no frame data is discarded, that is, to ensure that every frame data is processed. The central idea of the solution is very clear: let onPreviewFrame() return as soon as possible without discarding the frame data.

Here are four common processing methods: HandlerThread, Queue, AsyncTask and ThreadPool. The advantages and disadvantages of each method are simply analyzed.

HandlerThread

brief introduction

Using HandlerThread is to use Android's Message Queue to process frame data asynchronously. The simple process is to encapsulate the frame data as Message when onPreviewFrame() is called and send it to HandlerThread. HandlerThread obtains Message in the new thread and processes the frame data. Because it takes a short time to send a Message, it will not cause frame data loss.

Realization

Create a new ProcessWithHandlerThread class with the content of

Java

public class ProcessWithHandlerThread extends HandlerThread implements Handler.Callback {
    private static final String TAG = "HandlerThread";
    public static final int WHAT_PROCESS_FRAME = 1;

    public ProcessWithHandlerThread(String name) {
        super(name);
        start();

    }

    @Override
    public boolean handleMessage(Message msg) {
        switch (msg.what) {
            case WHAT_PROCESS_FRAME:
                byte[] frameData = (byte[]) msg.obj;
                processFrame(frameData);
                return true;
            default:
                return false;
        }
    }

    private void processFrame(byte[] frameData) {
        Log.i(TAG, "test");
    }
}

ProcessWithHandlerThread inherits HandlerThread and Handler.Callback interface, which implements handleMessage() method to process the obtained Message. The frame data is encapsulated in the obj attribute of the Message and marked with what. processFrame() deals with frame data, for example only.

Next, instantiate ProcessWithHandlerThread, bind interface, encapsulate frame data and send Message in CameraPreview.

Add a new member variable to CameraPreview

Java

private static final int PROCESS_WITH_HANDLER_THREAD = 1;

private int processType = PROCESS_WITH_HANDLER_THREAD;

private ProcessWithHandlerThread processFrameHandlerThread;
private Handler processFrameHandler;

Add at the end of the constructor

Java

switch (processType) {
    case PROCESS_WITH_HANDLER_THREAD:
        processFrameHandlerThread = new ProcessWithHandlerThread("process frame");
        processFrameHandler = new Handler(processFrameHandlerThread.getLooper(), processFrameHandlerThread);
        break;
}

Note that the new Handler() here is also binding the interface to have ProcessWithHandlerThread handle the received Message.

Modify onPreviewFrame() to

Java

public void onPreviewFrame(byte[] data, Camera camera) {
    switch (processType) {
        case PROCESS_WITH_HANDLER_THREAD:
            processFrameHandler.obtainMessage(ProcessWithHandlerThread.WHAT_PROCESS_FRAME, data).sendToTarget();
            break;
    }
}

Here, the frame data is encapsulated as Message and sent out.

Try running

Now run the APP, there will be a lot of "tests" in logcat, you can also modify processFrame() to test.

Analysis

This method is to flexibly apply Android's Handler mechanism and solve the problem with its Message Queue model, Message Queue. The problem is whether it will exceed the limit if all the frame data is encapsulated as Message and thrown to Message Queue. However, it has not been encountered yet. Another problem is that the Handler mechanism may be too large, which is not "lightweight" compared to dealing with this problem.

Queue

brief introduction

Queue method is to use queue to establish frame data queue. onPreviewFrame() is responsible for adding frame data to the end of the queue, while the processing method takes out frame data at the head of the queue and processes it. Queue is the role of buffering and providing interface.

Realization

Create a new ProcessWithQueue class with the content as

Java

public class ProcessWithQueue extends Thread {
    private static final String TAG = "Queue";
    private LinkedBlockingQueue<byte[]> mQueue;

    public ProcessWithQueue(LinkedBlockingQueue<byte[]> frameQueue) {
        mQueue = frameQueue;
        start();
    }

    @Override
    public void run() {
        while (true) {
            byte[] frameData = null;
            try {
                frameData = mQueue.take();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            processFrame(frameData);
        }
    }

    private void processFrame(byte[] frameData) {
        Log.i(TAG, "test");
    }
}

When ProcessWithQueue is instantiated, the Queue is provided externally. In order to process frame data independently and at any time, ProcessWithQueue inherits Thread and overloads run() method. The dead loop in the run() method is used to process the frame data in the Queue at any time. mQueue.take() blocks when the Queue is empty, so it will not cause CPU occupation caused by the loop. processFrame() deals with frame data, for example only.

Next, create a queue in CameraPreview and instantiate ProcessWithQueue to add frame data to the queue.

Add a new member variable to CameraPreview

Java

private static final int PROCESS_WITH_QUEUE = 2;

private ProcessWithQueue processFrameQueue;
private LinkedBlockingQueue<byte[]> frameQueue;

take

Java

private int processType = PROCESS_WITH_THREAD_POOL;

Modified to

Java

private int processType = PROCESS_WITH_QUEUE;

Add to the switch of the constructor

Java

case PROCESS_WITH_QUEUE:
    frameQueue = new LinkedBlockingQueue<>();
    processFrameQueue = new ProcessWithQueue(frameQueue);
    break;

The LinkedBlockingQueue is used to meet the concurrency requirements. Because only the head and tail of the team are operated, the linked list structure is used.

Add to the switch of onPreviewFrame()

Java

case PROCESS_WITH_QUEUE:
    try {
        frameQueue.put(data);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
    break;

Add frame data to the end of the queue.

Try running

Now run the APP, there will be a lot of "tests" in logcat, you can also modify processFrame() to test.

Analysis

This method can be simply understood as a simplification of the previous HandlerThread method. It only uses LinkedBlockingQueue to realize buffering, and writes out the queue processing method itself. This method also does not avoid the disadvantages mentioned before. If the frame data in the queue cannot be processed in time, it will cause the queue to be too long and occupy a lot of memory. But the advantage is simple and convenient.

AsyncTask

brief introduction

The AsyncTask method uses the AsyncTask class of Android, which will not be described in detail here. In short, each time AsyncTask is called, an asynchronous processing event will be created to execute the specified method asynchronously. In this case, the normal frame data processing method will be handed over to AsyncTask for execution.

Realization

Create a new ProcessWithAsyncTask class with the content of

Java

public class ProcessWithAsyncTask extends AsyncTask<byte[], Void, String> {
    private static final String TAG = "AsyncTask";

    @Override
    protected String doInBackground(byte[]... params) {
        processFrame(params[0]);
        return "test";
    }

    private void processFrame(byte[] frameData) {
        Log.i(TAG, "test");
    }
}

ProcessWithAsyncTask inherits AsyncTask, overloads doInBackground() method, and enters byte [] to return String. The code in doInBackground() is executed asynchronously. Here is processFrame(), which processes frame data. Here is just an example.

Next, you need to instantiate ProcessWithAsyncTask in CameraPreview and give the frame data to AsyncTask. Unlike the previous method, a new ProcessWithAsyncTask is instantiated and executed every time new frame data is processed.

Add a new member variable to CameraPreview

Java

private static final int PROCESS_WITH_ASYNC_TASK = 3;

take

Java

private int processType = PROCESS_WITH_QUEUE;

Modified to

Java

private int processType = PROCESS_WITH_ASYNC_TASK;

Add to the switch of onPreviewFrame()

Java

case PROCESS_WITH_ASYNC_TASK:
    new ProcessWithAsyncTask().execute(data);
    break;

Instantiate a new ProcessWithAsyncTask, pass frame data to it and execute it.

Try running

Now run the APP, there will be a lot of "tests" in logcat, you can also modify processFrame() to test.

Analysis

This method is simple in code, but it is difficult to understand its underlying implementation. AsyncTask actually uses thread pool technology to realize asynchronous and concurrent. Compared with the previous method, it has the advantage of high concurrency, but it can not go on indefinitely, and it will be restricted by the frame processing time. In addition, according to the introduction in the official documents, the emergence of AsyncTask is mainly to solve the problem of UI thread communication, so it's a sideshow here. AsyncTask has less "master" than the previous methods, and may not meet some requirements.

ThreadPool

brief introduction

The ThreadPool method mainly uses the Java ThreadPoolExecutor class. Presumably, the previous AsyncTask is a bit more low-level. The concurrent processing of frame data is realized by setting up thread pool manually.

Realization

Create a new ProcessWithThreadPool class with the content of

Java

public class ProcessWithThreadPool {
    private static final String TAG = "ThreadPool";
    private static final int KEEP_ALIVE_TIME = 10;
    private static final TimeUnit TIME_UNIT = TimeUnit.SECONDS;
    private BlockingQueue<Runnable> workQueue;
    private ThreadPoolExecutor mThreadPool;

    public ProcessWithThreadPool() {
        int corePoolSize = Runtime.getRuntime().availableProcessors();
        int maximumPoolSize = corePoolSize * 2;
        workQueue = new LinkedBlockingQueue<>();
        mThreadPool = new ThreadPoolExecutor(corePoolSize, maximumPoolSize, KEEP_ALIVE_TIME, TIME_UNIT, workQueue);
    }

    public synchronized void post(final byte[] frameData) {
        mThreadPool.execute(new Runnable() {
            @Override
            public void run() {
                processFrame(frameData);
            }
        });
    }

    private void processFrame(byte[] frameData) {
        Log.i(TAG, "test");
    }
}

The ProcessWithThreadPool constructor creates a thread pool. The corePoolSize is the concurrency, which is the number of processor cores. The maximum poolsize of the thread pool is set to twice the concurrency. post() is used to execute the frame data processing method through the thread pool. processFrame() deals with frame data, for example only.

Next, we will instantiate ProcessWithThreadPool in CameraPreview and give the frame data to ThreadPool.

Add a new member variable to CameraPreview

Java

private static final int PROCESS_WITH_THREAD_POOL = 4;

private ProcessWithThreadPool processFrameThreadPool;

take

Java

private int processType = PROCESS_WITH_ASYNC_TASK;

Modified to

Java

private int processType = PROCESS_WITH_THREAD_POOL;

Add to the switch of the constructor

Java

case PROCESS_WITH_THREAD_POOL:
    processFrameThreadPool = new ProcessWithThreadPool();
    break;

Add to the switch of onPreviewFrame()

Java

case PROCESS_WITH_THREAD_POOL:
    processFrameThreadPool.post(data);
    break;

Give the frame data to ThreadPool.

Try running

Now run the APP, there will be a lot of "tests" in logcat, you can also modify processFrame() to test.

Analysis

ThreadPool method is clearer than AsyncTask code, which is not "mysterious", but the two ideas are the same. ThreadPool method has more customization space when establishing thread pool, but it also fails to avoid the disadvantages of AsyncTask method.

A little nagging

Many of the methods described above only describe the idea of processing, and need to be modified according to the actual use, but the process is generally like this. Due to the lack of perfect test methods for real-time processing, bug s often exist and need to be carefully checked. For example, if two or three frames are lost in the processed frames, it is difficult to find out. Even if it is found, it is not easy to find out the wrong methods, and a large number of tests are needed.

The above methods are summarized based on the countless pits I stepped on. Because I haven't found a high-quality article about real-time preview frame processing, I can contribute some of my knowledge to help people in need even if I can achieve the goal.

For the actual processing problems of frame data and YUV format, please refer to some articles I wrote earlier about Android video decoding and YUV format parsing, and hope to help you.

DEMO

The camera APP source code implemented in this article is on GitHub. If you need to click zhantong/AndroidCamera-ProcessFrames.

Reference resources

Topics: Mobile Java Android xml SurfaceView

Programmer Think

(Reprint) Android camera development: efficient real-time processing of preview frame data

Summary

Get ready

Modify UI

Binding events

Try running

Basic frame data acquisition and processing

Basics

Try running

Separate from UI thread

problem analysis

Solution

Try running

Processing frame data

HandlerThread

brief introduction

Realization

Try running

Analysis

Queue

brief introduction

Realization

Try running

Analysis

AsyncTask

brief introduction

Realization

Try running

Analysis

ThreadPool

brief introduction

Realization

Try running

Analysis

A little nagging

DEMO

Reference resources

Hot Topics