Optimizing MediaMuxer’s writing speed.

Our efforts towards optimizing real-time video processing in Android.

If you targeting Android 4.3+ and you want to take advantage of the hardware video encoder, then you have to use MediaCodec for encoding video/audio frames into H.264/aac packets and MediaMuxer to write them in an mp4 format to disk. Grafika shows a plethora of scenarios where these 2 components are used. However, there is performance issue involved that we’ll discuss here.

One performance issue we faced on Horizon for Android was that MediaMuxer’s writeSampleData, would ocassionaly block for a couple of milliseconds to several seconds. The problem appeared on older devices and when writing to the SD card. In other words, when the medium’s writing speed was low, MediaMuxer’s buffering was not big enough to compensate for the delay and it would block until the data were actually written.

Before moving on with the solution, let’s first present a simple approach on feeding MediaCodec’s output to MediaMuxer:

This method is called when we drain MediaCodec’s output (usually in a dedicated thread). We have to call releaseOutputBuffer as soon as possible, so that MediaCodec can reuse this buffer to encode new packets. If muxer delays, MediaCodec will eventually ran out of buffers and when we will try to enqueue new video or audio packets on the other end, those methods will block repsectively: SwapBuffers will block, dequeueInputBuffer will block or return -1.

The solution

We have to copy the encoded data to a temporary ByteBuffer and call writeSampleData asynchronously on a dedicated thread:

This approach works. However, it requires that we allocate a new ByteBuffer for every incoming packet. Allocating direct buffers has bigger overhead than allocating a simple ByteBuffer. Also, we are hoping that the garbage collector will kick in at some point and reclaim the memory of the unused temporary byte buffers. Looking at ADB’s memory graph, it does.

However, since this method will run about 73 times/sec (for audio & video packets) it would be best if we could re-use the same memory somehow. The problem in implementing this approach, is that each video/audio packet has a different size. In order to make this work, we would have to allocate a large enough ByteBuffer that can hold a couple of seconds of encoded video/audio packets and write each new packet after the end of the previous one. Of course, we’ll have to do all the bookkeeping for the actual position of each packet in the big ByteBuffer.

Towards this approach, I used Grafika’s CircularBuffer and changed it so that it works for our case. You can find the resulting CircularBuffer here. First, we have to allocate the CircularBuffer using the MediaCodec’s MediaFormat:

mCircularBuffer = new CircularBuffer(trackFormat, 2000); // allocate 2 seconds buffer

This is how it’s used:

The above code, copies the incoming packet to the CircularBuffer and releases it almost instantly. Then, it asynchronosuly retrieves the packet from the CircularBuffer and feeds the MediaMuxer. There’s always the possibility that our CircularBuffer is full. In this case, the add() method returns -1 and we just block until enough room is made.

I have also created a increaseSize() method, that allocates more space on Circular’s buffer interal buffer. This way we don’t have to block. Here’s the updated code:

In the above code, when we can’t add a packet, we try to increase its size instead of blocking. If we fail to do that because we are out of memory, we just block. Tha’ts the best approach because we can start with a small internal buffer and -if the device is slow- we gradually increase it as needed.

All this, seems like a hard workaround to MediaMuxer’s bad buffering. Another approach would be using MediaRecorder. However, the methods needed to feed it with custom video frames (other than the camera’s) were added in Lollipop. So, if you want to support Android 4.3 and 4.4 you have to deal with this problem.

Originally published at blog.horizon.camera.