Android MediaCodec codec usage
Use MediaCodec for encoding and decoding. Input data in H.264 format, output frame data and send it to the listener.
Next, we call MediaCodec codec for short
H.264 configuration
Create and configure codec. When configuring codec, if you manually create MediaFormat objects, you must remember to set "csd-0" and "csd-1" parameters. "Csd-0" and "csd-1" must correspond to the received frame.
input data
When inputting data to codec, if you queue the input data, you need to check the queue.
For example, 1M memory is temporarily used for one frame of data and 30 frames in one second. The queue may temporarily use 30M memory. When the temporary memory usage is too high, we need to take some measures to reduce the memory occupation. codec hard decoding will be affected by mobile phone hardware. If the performance of the mobile phone is poor, the encoding and decoding speed may be slower than the original data input. If we have to, we can discard the old data in the queue and enter new data.
Decoder performance
codec has no available input buffer for scenes requiring high video real-time performance, mcodec Dequeueinputbuffer returns - 1. For real-time performance, the input / output buffer mcodec will be forcibly released here flush().
Problems and exceptions
Question 1 - is there a specific relationship between the amount of mediacodec input data and the amount of output data
For MediaCodec, is there a specific relationship between the amount of input data and the amount of output data? Suppose you input 10 frames of data, how many times can you get output?
It is found that the input and output times cannot be guaranteed to be equal. For example, vivo x6 plus, if you input 30 frames, you can get 28 frames of results. Or 300 inputs and 298 outputs.
Exception 1 - dequeueInputBuffer(0) always returns - 1
After long-time encoding and decoding of some mobile phones, the subscript may always return - 1 when trying to obtain the codec input buffer. For example, vivo x6 plus, after running for about 20 minutes, mcodec Dequeueinputbuffer (0) always returns - 1.
Processing method: if - 1 is always returned, try to call codec in synchronization mode Flush () method, and try codec in asynchronous mode After flush (), call codec Start() method.
Some mobile phones decode too slowly and may often return - 1. Do not call codec frequently Flush() to avoid abnormal display.
Code example - encoding and decoding in synchronous mode
This example uses synchronous encoding and decoding
Use synchronization mode
/**
* decoder
*/
public class CodecDecoder {
private static final String TAG = "CodecDecoder";
private static final String MIME_TYPE = "video/avc";
private static final String CSD0 = "csd-0";
private static final String CSD1 = "csd-1";
private static final int TIME_INTERNAL = 1;
private static final int DECODER_TIME_INTERNAL = 1;
private MediaCodec mCodec;
private long mCount = 0; // Media decoder for MediaCodec
// Buffer queue before sending to codec
// The temporary memory of this queue needs to be monitored in real time. If it is blocked here, it is easy to cause OOM
private Queue<byte[]> data = null;
private DecoderThread decoderThread;
private CodecListener listener; // A custom listener sends frame data through it when it is decoded
public CodecDecoder() {
data = new ConcurrentLinkedQueue<>();
}
public boolean isCodecCreated() {
return mCodec!=null;
}
public boolean createCodec(CodecListener listener, byte[] spsBuffer, byte[] ppsBuffer, int width, int height) {
this.listener = listener;
try {
mCodec = MediaCodec.createDecoderByType(Constants.MIME_TYPE);
MediaFormat mediaFormat = createVideoFormat(spsBuffer, ppsBuffer, width, height);
mCodec.configure(mediaFormat, null, null, 0);
mCodec.start();
Log.d(TAG, "decoderThread mediaFormat in:" + mediaFormat);
decoderThread = new DecoderThread();
decoderThread.start();
return true;
}
catch (Exception e) {
e.printStackTrace();
Log.e(TAG, "MediaCodec create error:" + e.getMessage());
return false;
}
}
private MediaFormat createVideoFormat(byte[] spsBuffer, byte[] ppsBuffer, int width, int height) {
MediaFormat mediaFormat;
mediaFormat = MediaFormat.createVideoFormat(MIME_TYPE, width, height);
mediaFormat.setByteBuffer(CSD0, ByteBuffer.wrap(spsBuffer));
mediaFormat.setByteBuffer(CSD1, ByteBuffer.wrap(ppsBuffer));
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
return mediaFormat;
}
private long lastInQueueTime = 0;
// Input H.264 frame data, and the queuing condition will be monitored here
public void addData(byte[] dataBuffer) {
final long timeDiff = System.currentTimeMillis() - lastInQueueTime;
if (timeDiff > 1) {
lastInQueueTime = System.currentTimeMillis();
int queueSize = data.size(); // The length of the ConcurrentLinkedQueue query will be traversed once, and this method is used as little as possible in the case of a large amount of data
if (queueSize > 30) {
data.clear();
LogInFile.getLogger().e("frame queue If the frame data queue exceeds the upper limit, the data will be cleared automatically " + queueSize);
}
data.add(dataBuffer.clone());
Log.e(TAG, "frame queue Add a frame of data");
} else {
LogInFile.getLogger().e("frame queue Adding too fast,Skip this frame. timeDiff=" + timeDiff);
}
}
public void destroyCodec() {
if (mCodec != null) {
try {
mCount = 0;
if(data!=null) {
data.clear();
data = null;
}
if(decoderThread!=null) {
decoderThread.stopThread();
decoderThread = null;
}
mCodec.release();
mCodec = null;
}
catch (Exception e) {
e.printStackTrace();
Log.d(TAG, "destroyCodec exception:" + e.toString());
}
}
}
private class DecoderThread extends Thread {
private final int INPUT_BUFFER_FULL_COUNT_MAX = 50;
private boolean isRunning;
private int inputBufferFullCount = 0; // How many times is the input buffer full
public void stopThread() {
isRunning = false;
}
@Override
public void run() {
setName("CodecDecoder_DecoderThread-" + getId());
isRunning = true;
while (isRunning) {
try {
if (data != null && !data.isEmpty()) {
int inputBufferIndex = mCodec.dequeueInputBuffer(0);
if (inputBufferIndex >= 0) {
byte[] buf = data.poll();
ByteBuffer inputBuffer = mCodec.getInputBuffer(inputBufferIndex);
if (null != inputBuffer) {
inputBuffer.clear();
inputBuffer.put(buf, 0, buf.length);
mCodec.queueInputBuffer(inputBufferIndex, 0,
buf.length, mCount * TIME_INTERNAL, 0);
mCount++;
}
inputBufferFullCount = 0; // There are also buffers that can be used to reset the count
} else {
inputBufferFullCount++;
LogInFile.getLogger().e(TAG, "decoderThread inputBuffer full. inputBufferFullCount=" + inputBufferFullCount);
if (inputBufferFullCount > INPUT_BUFFER_FULL_COUNT_MAX) {
mCount = 0;
mCodec.flush(); // Clear all buffers here
LogInFile.getLogger().e(TAG, "mCodec.flush()...");
}
}
}
// Get output buffer index
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = mCodec.dequeueOutputBuffer(bufferInfo, 0);
while (outputBufferIndex >= 0) {
final int index = outputBufferIndex;
Log.d(TAG, "releaseOutputBuffer " + Thread.currentThread().toString());
final ByteBuffer outputBuffer = byteBufferClone(mCodec.getOutputBuffer(index));
Image image = mCodec.getOutputImage(index);
if (null != image) {
// Get data in NV21 format
final byte[] nv21 = ImageUtil.getDataFromImage(image, FaceDetectUtil.COLOR_FormatNV21);
final int imageWid = image.getWidth();
final int imageHei = image.getHeight();
// Here you choose to create a new thread to send data - this is where you can optimize
new Thread(new Runnable() {
@Override
public void run() {
listener.onDataDecoded(outputBuffer,
mCodec.getOutputFormat().getInteger(MediaFormat.KEY_COLOR_FORMAT),
nv21, imageWid, imageHei);
}
}).start();
} else {
listener.onDataDecoded(outputBuffer,
mCodec.getOutputFormat().getInteger(MediaFormat.KEY_COLOR_FORMAT),
new byte[]{0}, 0, 0);
}
try {
mCodec.releaseOutputBuffer(index, false);
} catch (IllegalStateException ex) {
android.util.Log.e(TAG, "releaseOutputBuffer ERROR", ex);
}
outputBufferIndex = mCodec.dequeueOutputBuffer(bufferInfo, 0);
}
}
catch (Exception e) {
e.printStackTrace();
Log.e(TAG, "decoderThread exception:" + e.getMessage());
}
try {
Thread.sleep(DECODER_TIME_INTERNAL);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
// deep clone byteBuffer
private static ByteBuffer byteBufferClone(ByteBuffer buffer) {
if (buffer.remaining() == 0)
return ByteBuffer.wrap(new byte[]{0});
ByteBuffer clone = ByteBuffer.allocate(buffer.remaining());
if (buffer.hasArray()) {
System.arraycopy(buffer.array(), buffer.arrayOffset() + buffer.position(), clone.array(), 0, buffer.remaining());
} else {
clone.put(buffer.duplicate());
clone.flip();
}
return clone;
}
}Code example - tool functions
Some tool functions. For example, extract data in NV21 format from image.
Tool function
private byte[] getDataFromImage(Image image) {
return getDataFromImage(image, COLOR_FormatNV21);
}
/**
* According to the byte data of colorFormat type
*/
private byte[] getDataFromImage(Image image, int colorFormat) {
if (colorFormat != COLOR_FormatI420 && colorFormat != COLOR_FormatNV21) {
throw new IllegalArgumentException("only support COLOR_FormatI420 " + "and COLOR_FormatNV21");
}
if (!isImageFormatSupported(image)) {
throw new RuntimeException("can't convert Image to byte array, format " + image.getFormat());
}
Rect crop = image.getCropRect();
int format = image.getFormat();
int width = crop.width();
int height = crop.height();
Image.Plane[] planes = image.getPlanes();
byte[] data = new byte[width * height * ImageFormat.getBitsPerPixel(format) / 8];
byte[] rowData = new byte[planes[0].getRowStride()];
int channelOffset = 0;
int outputStride = 1;
for (int i = 0; i < planes.length; i++) {
switch (i) {
case 0:
channelOffset = 0;
outputStride = 1;
break;
case 1:
if (colorFormat == COLOR_FormatI420) {
channelOffset = width * height;
outputStride = 1;
} else if (colorFormat == COLOR_FormatNV21) {
channelOffset = width * height + 1;
outputStride = 2;
}
break;
case 2:
if (colorFormat == COLOR_FormatI420) {
channelOffset = (int) (width * height * 1.25);
outputStride = 1;
} else if (colorFormat == COLOR_FormatNV21) {
channelOffset = width * height;
outputStride = 2;
}
break;
}
ByteBuffer buffer = planes[i].getBuffer();
int rowStride = planes[i].getRowStride();
int pixelStride = planes[i].getPixelStride();
int shift = (i == 0) ? 0 : 1;
int w = width >> shift;
int h = height >> shift;
buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
for (int row = 0; row < h; row++) {
int length;
if (pixelStride == 1 && outputStride == 1) {
length = w;
buffer.get(data, channelOffset, length);
channelOffset += length;
} else {
length = (w - 1) * pixelStride + 1;
buffer.get(rowData, 0, length);
for (int col = 0; col < w; col++) {
data[channelOffset] = rowData[col * pixelStride];
channelOffset += outputStride;
}
}
if (row < h - 1) {
buffer.position(buffer.position() + rowStride - length);
}
}
}
return data;
}
/**
* Is it a supported data type
*/
private static boolean isImageFormatSupported(Image image) {
int format = image.getFormat();
switch (format) {
case ImageFormat.YUV_420_888:
case ImageFormat.NV21:
case ImageFormat.YV12:
return true;
}
return false;
}What are "csd-0" and "csd-1". For H264 video, it corresponds to sps and pps. For AAC audio, it corresponds to ADTS. People engaged in audio and video development should know that it generally exists in the IDR frame generated by the encoder.
Obtained mediaFormat
mediaFormat in:{height=720, width=1280, csd-1=java.nio.ByteArrayBuffer[position=0,limit=7,capacity=7], mime=video/avc, csd-0=java.nio.ByteArrayBuffer[position=0,limit=13,capacity=13], color-format=2135033992}Method of storing pictures
The Image class is very powerful in Android API21 and later.
Use the Image class to store pictures
private static void dumpFile(String fileName, byte[] data) {
FileOutputStream outStream;
try {
outStream = new FileOutputStream(fileName);
} catch (IOException ioe) {
throw new RuntimeException("rustfisher: Unable to create output file " + fileName, ioe);
}
try {
outStream.write(data);
outStream.close();
} catch (IOException ioe) {
throw new RuntimeException("rustfisher: failed writing data to file " + fileName, ioe);
}
}
private void compressToJpeg(String fileName, Image image) {
FileOutputStream outStream;
try {
outStream = new FileOutputStream(fileName);
} catch (IOException ioe) {
throw new RuntimeException("rustfisher: Unable to create output file " + fileName, ioe);
}
Rect rect = image.getCropRect();
YuvImage yuvImage = new YuvImage(getDataFromImage(image, COLOR_FormatNV21), ImageFormat.NV21, rect.width(), rect.height(), null);
yuvImage.compressToJpeg(rect, 100, outStream);
}Method of converting NV21 to bitmap
Direct storage in file
nv21 is saved as a jpg file
// in try catch FileOutputStream fos = new FileOutputStream(Environment.getExternalStorageDirectory() + "/rustfisher.jpg"); YuvImage yuvImage = new YuvImage(nv21bytearray, ImageFormat.NV21, width, height, null); yuvImage.compressToJpeg(new Rect(0, 0, width, height), 100, fos); fos.close();
The method to obtain the Bitmap object, which is time-consuming and memory consuming
NV21 -> yuvImage -> jpeg -> bitmap
// in try catch YuvImage yuvImage = new YuvImage(nv21bytearray, ImageFormat.NV21, width, height, null); ByteArrayOutputStream os = new ByteArrayOutputStream(); yuvImage.compressToJpeg(new Rect(0, 0, width, height), 100, os); byte[] jpegByteArray = os.toByteArray(); Bitmap bitmap = BitmapFactory.decodeByteArray(jpegByteArray, 0, jpegByteArray.length); os.close();
codec selects YUV420 format to output OutputBuffer
Suppose codec selects the format COLOR_FormatYUV420Flexible
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
After decoding, the format is COLOR_QCOM_FormatYUV420SemiPlanar32m // 0x7FA30C04
mCodec.getOutputFormat().getInteger(MediaFormat.KEY_COLOR_FORMAT)
Decoded ByteBuffer obuffer = mcodec getOutputBuffer(index); The number of elements included is 1413120; It is recorded as nv21Codec and passed through mcodec The number of nv21 array elements obtained by the image object obtained by getoutputimage (index) is 1382400; It's called nv21. These are the data we want
Comparing the two arrays, we find that the previous y part is the same. The first 921600 elements of nv21 are y data, and the last 460800 elements are uv data. The first 921600 elements of nv21Codec are y data, the next 20480 bytes are 0, and the next 460800 elements are uv data. The last 10240 bytes are 0
The storage order of the uv part of nv21 and nv21Codec is opposite.