I. Application screenshots

Preface

GPUImage is an open source image rendering library, which can easily achieve many filter effects, and also can easily define and implement its own unique filter effects.

Address: https://github.com/cats-oss/android-gpuimage

III. Dependent Engineering

To use GPUImage, Android Studio only needs to add dependencies to build.gradle.

implementation 'jp.co.cyberagent.android:gpuimage:2.0.3'

Some of the classes of GPUImage are introduced. After that, let's familiarize ourselves with the use of GPUImage.

Engineering Code

First of all, the project imitates Sample in GPUImage. Many codes are used for reference. The UI has been changed and some usage has been optimized.

1. Camera layout

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
              android:layout_width="match_parent"
              android:layout_height="match_parent"
              android:background="@android:color/black"
              android:orientation="vertical">

    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="horizontal">

        <ImageView
            android:id="@+id/close_iv"
            android:layout_width="40dp"
            android:layout_height="40dp"
            android:layout_marginLeft="10dp"
            android:padding="8dp"
            android:src="@mipmap/ic_close"/>

        <ImageView
            android:id="@+id/switch_camera_iv"
            android:layout_width="40dp"
            android:layout_height="40dp"
            android:layout_weight="1"
            android:padding="8dp"
            android:src="@mipmap/ic_switch_camera"/>

        <ImageView
            android:id="@+id/compare_iv"
            android:layout_width="40dp"
            android:layout_height="40dp"
            android:layout_marginRight="10dp"
            android:padding="8dp"
            android:src="@mipmap/ic_compare"/>
    </LinearLayout>

    <FrameLayout
        android:layout_width="match_parent"
        android:layout_height="0dp"
        android:layout_weight="1">

        <jp.co.cyberagent.android.gpuimage.GPUImageView
            android:id="@+id/gpuimage"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_gravity="center"/>

        <SeekBar
            android:id="@+id/tone_seekbar"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:layout_gravity="bottom"
            android:background="#55ffffff"
            android:max="100"
            android:padding="10dp"
            android:visibility="gone"/>

    </FrameLayout>

    <RelativeLayout
        android:layout_width="match_parent"
        android:layout_height="64dp"
        android:gravity="center_vertical"
        android:orientation="horizontal"
        android:padding="10dp">

        <TextView
            android:id="@+id/filter_name_tv"
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:layout_centerInParent="true"
            android:text="@string/choose_filter"
            android:textColor="@android:color/white"
            android:textSize="18sp"/>

        <ImageView
            android:id="@+id/save_iv"
            android:layout_width="36dp"
            android:layout_height="36dp"
            android:layout_alignParentEnd="true"
            android:layout_centerVertical="true"
            android:padding="5dp"
            android:src="@mipmap/ic_ok"/>
    </RelativeLayout>
</LinearLayout>

In addition to some conventional controls, we also use a custom control called GPUImageView as display, which is one of the most common classes we use GPUImage.

2. CameraActivity

public class CameraActivity extends BaseActivity implements View.OnClickListener {

    private GPUImageView mGPUImageView;
    private SeekBar mSeekBar;
    private TextView mFilterNameTv;

    private GPUImageFilter mNoImageFilter = new GPUImageFilter();
    private GPUImageFilter mCurrentImageFilter = mNoImageFilter;
    private GPUImageFilterTools.FilterAdjuster mFilterAdjuster;

    private CameraLoader mCameraLoader;

    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        initView();
        initCamera();
    }

    private void initView() {
        mGPUImageView = findViewById(R.id.gpuimage);
        mSeekBar = findViewById(R.id.tone_seekbar);
        mFilterNameTv = findViewById(R.id.filter_name_tv);
        mFilterNameTv.setOnClickListener(this);
        mSeekBar.setOnSeekBarChangeListener(mOnSeekBarChangeListener);
        findViewById(R.id.compare_iv).setOnTouchListener(mOnTouchListener);
        findViewById(R.id.close_iv).setOnClickListener(this);
        findViewById(R.id.save_iv).setOnClickListener(this);
        findViewById(R.id.switch_camera_iv).setOnClickListener(this);
    }

    private void initCamera() {
        mCameraLoader = new Camera2Loader(this);
        mCameraLoader.setOnPreviewFrameListener(new CameraLoader.OnPreviewFrameListener() {
            @Override
            public void onPreviewFrame(byte[] data, int width, int height) {
                mGPUImageView.updatePreviewFrame(data, width, height);
            }
        });
        mGPUImageView.setRatio(0.75f); // Fixed size 4:3
        updateGPUImageRotate();
        mGPUImageView.setRenderMode(GPUImageView.RENDERMODE_CONTINUOUSLY);
    }

    private void updateGPUImageRotate() {
        Rotation rotation = getRotation(mCameraLoader.getCameraOrientation());
        boolean flipHorizontal = false;
        boolean flipVertical = false;
        if (mCameraLoader.isFrontCamera()) { // Front-facing cameras need mirroring
            if (rotation == Rotation.NORMAL || rotation == Rotation.ROTATION_180) {
                flipHorizontal = true;
            } else {
                flipVertical = true;
            }
        }
        mGPUImageView.getGPUImage().setRotation(rotation, flipHorizontal, flipVertical);
    }

    @Override
    protected void onResume() {
        super.onResume();
        if (ViewCompat.isLaidOut(mGPUImageView) && !mGPUImageView.isLayoutRequested()) {
            mCameraLoader.onResume(mGPUImageView.getWidth(), mGPUImageView.getHeight());
        } else {
            mGPUImageView.addOnLayoutChangeListener(new View.OnLayoutChangeListener() {
                @Override
                public void onLayoutChange(View v, int left, int top, int right, int bottom, int oldLeft, int oldTop,
                                           int oldRight, int oldBottom) {
                    mGPUImageView.removeOnLayoutChangeListener(this);
                    mCameraLoader.onResume(mGPUImageView.getWidth(), mGPUImageView.getHeight());
                }
            });
        }
    }

    @Override
    protected void onPause() {
        super.onPause();
        mCameraLoader.onPause();
    }

    @Override
    public void onClick(View v) {
        switch (v.getId()) {
            case R.id.filter_name_tv:
                GPUImageFilterTools.showDialog(this, mOnGpuImageFilterChosenListener);
                break;
            case R.id.close_iv:
                finish();
                break;
            case R.id.save_iv:
                saveSnapshot();
                break;
            case R.id.switch_camera_iv:
                mGPUImageView.getGPUImage().deleteImage();
                mCameraLoader.switchCamera();
                updateGPUImageRotate();
                break;
        }
    }

    private void saveSnapshot() {
        String fileName = System.currentTimeMillis() + ".jpg";
        mGPUImageView.saveToPictures("GPUImage", fileName, mOnPictureSavedListener);
    }

    private View.OnTouchListener mOnTouchListener = new View.OnTouchListener() {
        @Override
        public boolean onTouch(View v, MotionEvent event) {
            if (v.getId() == R.id.compare_iv) {
                switch (event.getAction()) {
                    case MotionEvent.ACTION_DOWN:
                        mGPUImageView.setFilter(mNoImageFilter);
                        break;
                    case MotionEvent.ACTION_UP:
                        mGPUImageView.setFilter(mCurrentImageFilter);
                        break;
                }
            }
            return true;
        }
    };

    private OnGpuImageFilterChosenListener mOnGpuImageFilterChosenListener = new OnGpuImageFilterChosenListener() {
        @Override
        public void onGpuImageFilterChosenListener(GPUImageFilter filter, String filterName) {
            switchFilterTo(filter);
            mFilterNameTv.setText(filterName);
        }
    };

    private void switchFilterTo(GPUImageFilter filter) {
        if (mCurrentImageFilter == null
                || (filter != null && !mCurrentImageFilter.getClass().equals(filter.getClass()))) {
            mCurrentImageFilter = filter;
            mGPUImageView.setFilter(mCurrentImageFilter);
            mFilterAdjuster = new GPUImageFilterTools.FilterAdjuster(mCurrentImageFilter);
            mSeekBar.setVisibility(mFilterAdjuster.canAdjust() ? View.VISIBLE : View.GONE);
        } else {
            mSeekBar.setVisibility(View.GONE);
        }
    }

    private SeekBar.OnSeekBarChangeListener mOnSeekBarChangeListener = new SeekBar.OnSeekBarChangeListener() {
        @Override
        public void onProgressChanged(SeekBar seekBar, int progress, boolean fromUser) {
            if (mFilterAdjuster != null) {
                mFilterAdjuster.adjust(progress);
            }
            mGPUImageView.requestRender();
        }
        @Override
        public void onStartTrackingTouch(SeekBar seekBar) {}
        @Override
        public void onStopTrackingTouch(SeekBar seekBar) {}
    };

    private GPUImageView.OnPictureSavedListener mOnPictureSavedListener = new GPUImageView.OnPictureSavedListener() {
        @Override
        public void onPictureSaved(Uri uri) {
            String filePath = FileUtils.getRealFilePath(CameraActivity.this, uri);
            Log.d(TAG, "save to " + filePath);
            Toast.makeText(CameraActivity.this, "Saved: " + filePath, Toast.LENGTH_SHORT).show();
        }
    };

    private Rotation getRotation(int orientation) {
        switch (orientation) {
            case 90:
                return Rotation.ROTATION_90;
            case 180:
                return Rotation.ROTATION_180;
            case 270:
                return Rotation.ROTATION_270;
            default:
                return Rotation.NORMAL;
        }
    }
}

Apart from some events, it is mainly through Camera Loader's OnPreview FrameListener callback to get frame data and update it.

Moreover, we only need to switch different GPUI mageFilters to achieve different filter effects, which is very convenient.

3. CameraLoader

Camera operation class, Abstract class, ready for Camera1 that may need to be used.

public abstract class CameraLoader {

    protected OnPreviewFrameListener mOnPreviewFrameListener;

    public abstract void onResume(int width, int height);

    public abstract void onPause();

    public abstract void switchCamera();

    public abstract int getCameraOrientation();

    public abstract boolean hasMultipleCamera();

    public abstract boolean isFrontCamera();

    public void setOnPreviewFrameListener(OnPreviewFrameListener onPreviewFrameListener) {
        mOnPreviewFrameListener = onPreviewFrameListener;
    }

    public interface OnPreviewFrameListener {
        void onPreviewFrame(byte[] data, int width, int height);
    }

}

4. Camera2Loader

It inherits from Camera Loader and uses the relevant API of Camera 2 to complete the operation of the camera.

public class Camera2Loader extends CameraLoader {

    private static final String TAG = "Camera2Loader";

    private Activity mActivity;

    private CameraManager mCameraManager;
    private CameraCharacteristics mCharacteristics;
    private CameraDevice mCameraDevice;
    private CameraCaptureSession mCaptureSession;
    private ImageReader mImageReader;

    private String mCameraId;
    private int mCameraFacing = CameraCharacteristics.LENS_FACING_BACK;
    private int mViewWidth;
    private int mViewHeight;
    private float mAspectRatio = 0.75f; // 4:3

    public Camera2Loader(Activity activity) {
        mActivity = activity;
        mCameraManager = (CameraManager) mActivity.getSystemService(Context.CAMERA_SERVICE);
    }

    @Override
    public void onResume(int width, int height) {
        mViewWidth = width;
        mViewHeight = height;
        setUpCamera();
    }

    @Override
    public void onPause() {
        releaseCamera();
    }

    @Override
    public void switchCamera() {
        mCameraFacing ^= 1;
        Log.d(TAG, "current camera facing is: " + mCameraFacing);
        releaseCamera();
        setUpCamera();
    }

    @Override
    public int getCameraOrientation() {
        int degrees = mActivity.getWindowManager().getDefaultDisplay().getRotation();
        switch (degrees) {
            case Surface.ROTATION_0:
                degrees = 0;
                break;
            case Surface.ROTATION_90:
                degrees = 90;
                break;
            case Surface.ROTATION_180:
                degrees = 180;
                break;
            case Surface.ROTATION_270:
                degrees = 270;
                break;
            default:
                degrees = 0;
                break;
        }
        int orientation = 0;
        try {
            String cameraId = getCameraId(mCameraFacing);
            CameraCharacteristics characteristics = mCameraManager.getCameraCharacteristics(cameraId);
            orientation = characteristics.get(CameraCharacteristics.SENSOR_ORIENTATION);
        } catch (CameraAccessException e) {
            e.printStackTrace();
        }
        Log.d(TAG, "degrees: " + degrees + ", orientation: " + orientation + ", mCameraFacing: " + mCameraFacing);
        if (mCameraFacing == CameraCharacteristics.LENS_FACING_FRONT) {
            return (orientation + degrees) % 360;
        } else {
            return (orientation - degrees) % 360;
        }
    }

    @Override
    public boolean hasMultipleCamera() {
        try {
            int size = mCameraManager.getCameraIdList().length;
            return size > 1;
        } catch (CameraAccessException e) {
            e.printStackTrace();
        }
        return false;
    }

    @Override
    public boolean isFrontCamera() {
        return mCameraFacing == CameraCharacteristics.LENS_FACING_FRONT;
    }

    @SuppressLint("MissingPermission")
    private void setUpCamera() {
        try {
            mCameraId = getCameraId(mCameraFacing);
            mCharacteristics = mCameraManager.getCameraCharacteristics(mCameraId);
            mCameraManager.openCamera(mCameraId, mCameraDeviceCallback, null);
        } catch (CameraAccessException e) {
            Log.e(TAG, "Opening camera (ID: " + mCameraId + ") failed.");
            e.printStackTrace();
        }
    }

    private void releaseCamera() {
        if (mCaptureSession != null) {
            mCaptureSession.close();
            mCaptureSession = null;
        }
        if (mCameraDevice != null) {
            mCameraDevice.close();
            mCameraDevice = null;
        }
        if (mImageReader != null) {
            mImageReader.close();
            mImageReader = null;
        }
    }

    private String getCameraId(int facing) throws CameraAccessException {
        for (String cameraId : mCameraManager.getCameraIdList()) {
            if (mCameraManager.getCameraCharacteristics(cameraId).get(CameraCharacteristics.LENS_FACING) ==
                    facing) {
                return cameraId;
            }
        }
        // default return
        return Integer.toString(facing);
    }

    private void startCaptureSession() {
        Size size = chooseOptimalSize();
        Log.d(TAG, "size: " + size.toString());
        mImageReader = ImageReader.newInstance(size.getWidth(), size.getHeight(), ImageFormat.YUV_420_888, 2);
        mImageReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
            @Override
            public void onImageAvailable(ImageReader reader) {
                if (reader != null) {
                    Image image = reader.acquireNextImage();
                    if (image != null) {
                        if (mOnPreviewFrameListener != null) {
                            byte[] data = ImageUtils.generateNV21Data(image);
                            mOnPreviewFrameListener.onPreviewFrame(data, image.getWidth(), image.getHeight());
                        }
                        image.close();
                    }
                }
            }
        }, null);
        try {
            mCameraDevice.createCaptureSession(Arrays.asList(mImageReader.getSurface()), mCaptureStateCallback, null);
        } catch (CameraAccessException e) {
            e.printStackTrace();
            Log.e(TAG, "Failed to start camera session");
        }
    }

    private Size chooseOptimalSize() {
        Log.d(TAG, "viewWidth: " + mViewWidth + ", viewHeight: " + mViewHeight);
        if (mViewWidth == 0 || mViewHeight == 0) {
            return new Size(0, 0);
        }
        StreamConfigurationMap map = mCharacteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
        Size[] sizes = map.getOutputSizes(ImageFormat.YUV_420_888);
        int orientation = getCameraOrientation();
        boolean swapRotation = orientation == 90 || orientation == 270;
        int width = swapRotation ? mViewHeight : mViewWidth;
        int height = swapRotation ? mViewWidth : mViewHeight;
        return getSuitableSize(sizes, width, height, mAspectRatio);
    }

    private Size getSuitableSize(Size[] sizes, int width, int height, float aspectRatio) {
        int minDelta = Integer.MAX_VALUE;
        int index = 0;
        Log.d(TAG, "getSuitableSize. aspectRatio: " + aspectRatio);
        for (int i = 0; i < sizes.length; i++) {
            Size size = sizes[i];
            // First judge whether the proportions are equal
            if (size.getWidth() * aspectRatio == size.getHeight()) {
                int delta = Math.abs(width - size.getWidth());
                if (delta == 0) {
                    return size;
                }
                if (minDelta > delta) {
                    minDelta = delta;
                    index = i;
                }
            }
        }
        return sizes[index];
    }

    private CameraDevice.StateCallback mCameraDeviceCallback = new CameraDevice.StateCallback() {
        @Override
        public void onOpened(@NonNull CameraDevice camera) {
            mCameraDevice = camera;
            startCaptureSession();
        }
        @Override
        public void onDisconnected(@NonNull CameraDevice camera) {
            mCameraDevice.close();
            mCameraDevice = null;
        }
        @Override
        public void onError(@NonNull CameraDevice camera, int error) {
            mCameraDevice.close();
            mCameraDevice = null;
        }
    };

    private CameraCaptureSession.StateCallback mCaptureStateCallback = new CameraCaptureSession.StateCallback() {
        @Override
        public void onConfigured(@NonNull CameraCaptureSession session) {
            if (mCameraDevice == null) {
                return;
            }
            mCaptureSession = session;
            try {
                CaptureRequest.Builder builder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
                builder.addTarget(mImageReader.getSurface());
                builder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
                builder.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH);
                session.setRepeatingRequest(builder.build(), null, null);
            } catch (CameraAccessException e) {
                e.printStackTrace();
            }
        }
        @Override
        public void onConfigureFailed(@NonNull CameraCaptureSession session) {
            Log.e(TAG, "Failed to configure capture session.");
        }
    };

As for the use of cameras, there are many things that can be optimized, such as simplified examples (for example, the choice of resolution only takes into account the proportion of 4:3, does not use background threads to perform some tasks, and some parameters of the camera are not set too much).

5. Other Categories

The rest are tools. You can find them in the project address.

5. Engineering Address

The following is the complete engineering code, which can be run directly on Android Studio.

https://github.com/afei-cn/GPUImageDemo

Introduction of GPUImage Class

1. Directory structure

| Filter: Below this package are various types of filter effects.
| - util: Below this package are some tool classes.
| GLTextureView: Inherited from TextureView, similar to GLSurfaceView.
| GPUImage: Core implementation class, rendering with GLSurfaceView/GLTextureView and GPUImageFilter.
| GPUImageNative Library: Contains some native methods for image transcoding.
| GPUImageRenderer: The actual renderer.
| GPUImageView: Inherited from FrameLayout, it encapsulates a GPUImage and GPUImageFilter, which is more convenient to use.

2. Brief use

GPUImage:

@Override
public void onCreate(final Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity);

    Uri imageUri = ...;
    gpuImage = new GPUImage(this);
    gpuImage.setGLSurfaceView((GLSurfaceView) findViewById(R.id.surfaceView));
    gpuImage.setImage(imageUri); // this loads image on the current thread, should be run in a thread
    gpuImage.setFilter(new GPUImageSepiaFilter());

    // Later when image should be saved saved:
    gpuImage.saveToPictures("GPUImage", "ImageWithFilter.jpg", null);
}

GPUImageView:

<jp.co.cyberagent.android.gpuimage.GPUImageView
    android:id="@+id/gpuimageview"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    app:gpuimage_show_loading="false"
    app:gpuimage_surface_type="texture_view" /> <!-- surface_view or texture_view -->

@Override
public void onCreate(final Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity);

    Uri imageUri = ...;
    gpuImageView = findViewById(R.id.gpuimageview);
    gpuImageView.setImage(imageUri); // this loads image on the current thread, should be run in a thread
    gpuImageView.setFilter(new GPUImageSepiaFilter());

    // Later when image should be saved saved:
    gpuImageView.saveToPictures("GPUImage", "ImageWithFilter.jpg", null);
}