Understanding the Android Multimedia Framework: From AwesomePlayer to OMX Service

Introduction: The previous article “Summary of Android Multimedia Framework (Part 10)” summarized the audio and video output process. Starting today, we will analyze the Codec part, focusing on the process from AwesomePlayer to the OMX service, which involves preparing for OpenMax.

Let’s take a look at today’s agenda:

• A diagram to clarify the position of OMX in Stagefright

• A diagram to clarify the relationship between OpenMax and Stagefright layers

• OMX initialization process

• Management of NodeInstance list in OMX service

• Operations on NodeInstance nodes in OMX service

• Summary of the process from AwesomePlayer to OMX service

Background: In the Android system, OpenMAX is used for encoding and decoding. Android abstracts a layer called OMXCodec for the upper layer player AwesomePlayer. In the player, the audio and video decoders mVideosource and mAudiosource are instances of OMXCodec. OMXCodec::Create is the entry point for decoder initialization. OMXCodec obtains the OMX service through the IOMX interface, which relies on the binder mechanism, and the OMX service is the implementation of OpenMAX in Android.

A diagram to clarify the position of OMX in Stagefright:

A diagram to clarify the relationship between OpenMax and Stagefright layers:

OMX Initialization Process

How does AwesomePlayer obtain the OMX service?

• During the initialization of AwesomePlayer, it calls AwesomePlayer::onPrepareAsyncEvent.

• Then it calls AwesomePlayer::initVideoDecoder and AwesomePlayer::initAudioDecoder.

• Next, it formally enters the initialization work of OMX and the hardware decoder.

Prior to this, some of the initialization work of AwesomePlayer was minor. When OMX starts initializing, it marks the beginning of the core initialization work. We know that components in Android are service providers, there are servers and clients, mostly in a C/S model, as introduced in previous articles. AwesomePlayer has a variable OMXClient mClient; let’s first understand OMXClient.cpp:

OMXClient has an IOMX variable mOMX, which communicates with the OMX service via the binder.

In AwesomePlayer’s constructor, it calls: CHECK_EQ(mClient.connect(), (status_t)OK); The code in OMXClient is as follows:

OMXClient::connect function obtains the MediaPlayerService through the binder mechanism, and then creates the OMX instance through MediaPlayerService. Thus, OMXClient obtains the entry point to OMX, and can then use the binder mechanism to obtain the services provided by OMX. In other words, OMXClient is the entry point for OpenMAX in Android.

When creating audio and video decoders mVideoSource and mAudioSource, the instance of sp mOMX in OMXClient is passed to mVideoSource and mAudioSource for shared usage. This means that one AwesomePlayer corresponds to one IOMX variable, and the audio and video decoders in AwesomePlayer share this IOMX variable to obtain the OMX service.

The above is divided into two steps; let’s first look at the OMXCodec::Create function:

Each AwesomePlayer instance has only one entry point for the OMX service, but AwesomePlayer may require more than one type of decoder. Both audio and video are needed, and in some scenarios, there may be multiple audio or video streams. In this case, different decoder components need to be established on the OMX side to correspond to the different decoding requirements of AwesomePlayer.

This article is from Yuiop: http://blog.csdn.net/hejjunlin/article/details/52623882

Two very important members in OMX: OMXMaster and OMXNodeInstance.

• OMX creates and maintains different OpenMAX decoder components through these two members, providing services for the different decoding needs in AwesomePlayer. OMXNodeInstance is responsible for creating and maintaining different instances, which are created based on actual decoding requirements, identified uniquely by node_id.

• Thus, each OMXCodec in the decoder component corresponds to its own OMXNodeInstance instance on the OMX server. AwesomePlayer can operate the corresponding decoder based on this OMXNodeInstance.

• OMXMaster maintains the underlying software and hardware decoding libraries, acting as a manager for the decoder components, creating decoding entity components based on the decoders desired in OMXNodeInstance.

Therefore, we need to track OMXMaster and OMXNodeInstance. Initialization occurs in the constructor of OMX. In OMX.cpp:

In OMXMaster.cpp:

At this point, we understand how AwesomePlayer utilizes the hardware decoder on the specific hardware platform.

So how does it choose the specific decoder component for different file formats? Let’s continue with the introduction of OMXCodec::Create. Check the allocateNode in OMX.cpp:

It will call the makeComponentInstance function:

This connects the specific decoder based on the file encoding format. Next, let’s understand how OMXCodec registers and initializes the callback functions needed for OMX. The OMX service mainly completes three tasks: management of the NodeInstance list, operations on NodeInstance, and event handling.

This article is from Yuiop: http://blog.csdn.net/hejjunlin/article/details/52623882

Management of NodeInstance List

• OMX uses OMXNodeInstance to interact with the decoder component. The actions of OMXNodeInstance include generating (allocateNode) and deleting (freeNode) NodeInstances. This is essentially adding and removing from mDispatchers and mNodeIDToInstance.

• mNodeIDToInstance is a key-value pair list where the key is node_id and the value is NodeInstance. mDispatchers is a key-value pair list where the key is node_id and the value is OMX::CallbackDispatcher. Each NodeInstance has an OMX::CallbackDispatcher.

• The role of CallbackDispatcher is mainly to distribute messages to the corresponding OMXCodec client after the decoder component issues a callback action.

Operations on NodeInstance Nodes

The main member functions of OMXNodeInstance are as follows:

When these methods execute, they first find the corresponding NodeInstance in the mNodeIDToInstance list through findInstance, and then call the method for that NodeInstance.

• OMXCodec operates on the specific component methods through OMXNodeInstance. Methods like fillBuffer, emptyBuffer, and sendCommand are indirectly called through macro definitions in OMX_Core.h and completed using the corresponding function pointers in the struct OMX_COMPONENTTYPE in OMX_Component.h. Both OMX_Core.h and OMX_Component.h are OpenMAX standard header files. In OMXNodeInstance.cpp, there is a piece of code:

• It registers the three static methods of the OMXNodeInstance class to kCallbacks.

• kCallbacks is the specific implementation of struct OMX_COMPONENTTYPE and struct OMX_CALLBACKTYPE.

• These two are defined in OMX_Core.h and OMX_Component.h.

Where is kCallbacks used? Let’s look at the code in the allocateNode method in OMX.cpp:

The event handling function is passed to the component Instance. When an event occurs in the component, it will call the registered event handling functions in OMXNodeInstance:

These functions will then call the corresponding functions in OMX, which are the following three:

Summarizing the above code:

• These functions all adopt the same approach:

• Find the CallbackDispatcher based on node_id and post the event information. That is: findDispatcher(node)->post(msg).

Furthermore, we need to understand the implementation mechanism of CallbackDispatcher. It starts a thread internally and uses semaphore mechanisms (signal). findDispatcher(node)->post(msg) is an asynchronous operation that only posts the msg without waiting for the event handling to complete.

The question arises, how does CallbackDispatcher handle the received msg? Check the following code:

Thus, the event is ultimately passed across the Binder to the OMXCodec, handed over to the OMXCodecObserver. This means that the callback occurrence is sent from the service side to the client side.

Finally, let’s summarize the previous points:

• 1. During the initialization of AwesomePlayer, video/audio decoders mVideoSource/mAudioSource are created through initVideoDecoder / initAudioDecoder.

• 2. In mVideoSource, mVideoTrack demuxes the media file to obtain the file encoding format, which in turn determines the required decoder type, invoking omx->allocateNode to create the OMX node instance corresponding to the encoding format. All subsequent operations are done through the node instance to control the actual hardware decoder.

• 3. When initializing the MediaPlayerService object, it creates the OMX object. The constructor of the OMX object creates mMaster, which is responsible for obtaining and managing the hardware decoder component library of the hardware platform.

• 4. In omx->allocateNode, the real corresponding decoder component is created through mMaster->makeComponentInstance. This decoder component is responsible for the actual decoding operations.

• 5. During the creation of mMaster->makeComponentInstance, it also uses the decoder type name passed from mVideoTrack to find the corresponding decoder library and instantiate it.

• 6. The decoding component interacts through input and output ports, and performs encoding and decoding by sharing buffers with OMXCodec.

• 7. AwesomePlayer contains mVideoSource, which points to the actual object of OMXCodec during initialization. OMXCodec uses the Binder mechanism to achieve remote calls to the OMX service, where IOMX defines most of the interface functions for OMX.

• 8. The specific implementation of OMX is managed by the OMXMaster class, while the OMXNodeInstance class represents the specific instance of OMX, completing calls and interactions with the component.

• 9. CallbackDispatcher is used to schedule and handle messages returned from callback functions. OMXNodeInstance and CallbackDispatcher work together to process messages from different instances.

• 10. OMXNodeInstance is a concept on the OMX side, representing the service side. Its service side exists within the same process space as OMX.

• 11. OMXObserver is a concept on the OMXCodec side, representing the client side. Its service side exists within the same process space as OMXCodec, and its Bn and Bp directions are opposite to those of OMX and OMXNodeInstance. It is mainly used to notify onMessage messages in reverse. Now we have introduced how AwesomePlayer initializes OMX and associates it with the corresponding hardware platform’s HW decoder.

Get blog update notifications in real-time, and more Android content and source code analysis. Follow my WeChat public account by scanning the QR code below or long-pressing to identify the QR code to follow.