This article was reproduced from the sheldon_blogs blog at https://www.cnblogs.com/blogs-of-lxl/p/10651611.html
 This article is for study and research purposes only. Please attach the original author's name and web address if you want to reproduce it

1. Treble mechanism on Android O

In Android O, when the system starts, a CameraProvider service is started, which is separated from the cameraserver process and used to control the camera HAL as an independent process, android.hardware.camera.provider@2.4-service, which communicates with the camera provider through the HIDL mechanism.HIDL originates from the Treble mechanism added by Android O. Its main function is to isolate service from HAL for easy upgrade of HAL parts independently, similar to the Binder communication mechanism between APP and Framework, and decouple different levels by introducing an inter-process communication mechanism (from Local call to Remote call).As follows:

The calling logic for the initiation and initialization of two processes, cameraserver and provider, is as follows:

2. Framework update for Camera HAL3

• **Application framework:** Provides APP access to hardware's Camera API2 and camera service via binder.

• AIDL: A Binder-based interface for enabling App fw code to access natice fw code.Its implementation exists in the following paths: frameworks/av/camera/aidl/android/hardware.Where:

  (1) ICameraService is the interface of the camera service.Used to request connections, add monitoring, etc.
  (2) ICameraDeviceUser is the interface for a specific camera device that has been opened.It allows the application framework to access specific devices.
  (3) ICameraServiceListener and ICameraDeviceCallbacks are callbacks from CameraService and CameraDevice to the application framework, respectively.

• Natice framework: frameworks/av/.Provides the implementation of aidl interfaces such as ICameraService, ICameraDeviceUser, ICameraDeviceCallbacks, ICameraServiceListener.And camera server's main function.

• Binder IPC interface: Provides an interface for interprocess communication, communication between APP and CameraService, and communication between HAL.AIDL and HIDL are implemented based on Binder.

• Camera Service: frameworks/av/services/camera/.Services that interact with APP and HAL play a connecting role.

• **HAL:**The HAL of Google defines a standard interface that Camera Service can access.For vendors, these interfaces must be implemented.

1.Camera HAL3 Building Connections

The following figure (the red dashed line is the ascending route and the black dashed line is the descending route):

2. Call process from App to CameraService

Connecting from Application to CameraService involves three levels in the Android architecture: App, Framework, and Runtime.The App layer directly calls the methods encapsulated by the Framework layer, and the Framework layer needs to call CameraService's functions in Runtime remotely through Binder.
The main function call logic in this section is as follows:

In App, you need to call the API to open the camera as follows:

• CameraCharacteristics: Describes the various characteristics of the camera, which can be obtained by using CameraManager's getCameraCharacteristics(@NonNull String cameraId).
• CameraDevice: Describes the system Camera, similar to earlier Cameras.
• CameraCaptureSession: Session class. When you need to take a photo, preview, and so on, you need to create an instance of the class and then control it by the method in the instance (for example, photo capture()).
• CaptureRequest: Describes an operation request, which requires the CaptureRequest parameter to be passed in first for taking a photo, previewing, etc. The specific parameter control is also set through the member variables of CameraRequest.
• CaptureResult: Describes the result after taking a picture.

For example, open the java code for camera:

　mCameraManager.openCamera(currentCameraId, stateCallback, backgroundHandler);

The Camera2 process is as follows:

(1)Framework CameraManager

/frameworks/base/core/java/android/hardware/camera2/CameraManager.java

The initial entry is CameraManager's openCamera method, but as you can see from the code, it simply calls the openCameraForUid method.

@RequiresPermission(android.Manifest.permission.CAMERA)
public void openCamera(@NonNull String cameraId,
        @NonNull final CameraDevice.StateCallback callback, @Nullable Handler handler)
        throws CameraAccessException {

    openCameraForUid(cameraId, callback, handler, USE_CALLING_UID);
}

The code below omits some parameter checking and ultimately calls the openCameraDeviceUserAsync method.

public void openCameraForUid(@NonNull String cameraId,
        @NonNull final CameraDevice.StateCallback callback, @Nullable Handler handler,
        int clientUid)
        throws CameraAccessException {
    /* Do something in*/
    ......
    /* Do something out*/
    openCameraDeviceUserAsync(cameraId, callback, handler, clientUid);
}

Refer to the following notes for analysis:

private CameraDevice openCameraDeviceUserAsync(String cameraId,
        CameraDevice.StateCallback callback, Handler handler, final int uid)
        throws CameraAccessException {
    CameraCharacteristics characteristics = getCameraCharacteristics(cameraId);
    CameraDevice device = null;

    synchronized (mLock) {

        ICameraDeviceUser cameraUser = null;

        android.hardware.camera2.impl.CameraDeviceImpl deviceImpl =   //Instantiate a CameraDeviceImpl.CameraDevice.StateCallback and Handler are passed in during construction.
                new android.hardware.camera2.impl.CameraDeviceImpl(  
                    cameraId,
                    callback,
                    handler,
                    characteristics,
                    mContext.getApplicationInfo().targetSdkVersion);

        ICameraDeviceCallbacks callbacks = deviceImpl.getCallbacks(); //Gets the CameraDeviceCallback instance, which is the interface provided to the remote to connect to CameraDeviceImpl.

       try {
            if (supportsCamera2ApiLocked(cameraId)) {  //HAL3 takes this part of the logic, mainly by obtaining the local interface of CameraService from CameraManagerGlobal and connecting to the camera device through its remote call (using the Binder mechanism) connectDevice method.
　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　//Note that the returned cameraUser actually points to the local interface of the remote CameraDeviceClient.
                // Use cameraservice's cameradeviceclient implementation for HAL3.2+ devices
                ICameraService cameraService = CameraManagerGlobal.get().getCameraService();
                if (cameraService == null) {
                    throw new ServiceSpecificException(
                        ICameraService.ERROR_DISCONNECTED,
                        "Camera service is currently unavailable");
                }
                cameraUser = cameraService.connectDevice(callbacks, cameraId,
                        mContext.getOpPackageName(), uid);
            } else {
                // Use legacy camera implementation for HAL1 devices
                int id;
                try {
                    id = Integer.parseInt(cameraId);
                } catch (NumberFormatException e) {
                    throw new IllegalArgumentException("Expected cameraId to be numeric, but it was: "
                            + cameraId);
                }

                Log.i(TAG, "Using legacy camera HAL.");
                cameraUser = CameraDeviceUserShim.connectBinderShim(callbacks, id);
            }
        } catch (ServiceSpecificException e) {
            /* Do something in */
            ......
            /* Do something out */
        }

        // TODO: factor out callback to be non-nested, then move setter to constructor
        // For now, calling setRemoteDevice will fire initial
        // onOpened/onUnconfigured callbacks.
        // This function call may post onDisconnected and throw CAMERA_DISCONNECTED if
        // cameraUser dies during setup.
        deviceImpl.setRemoteDevice(cameraUser); //Set the CameraDeviceClient to CameraDeviceImpl for management.
        device = deviceImpl;
    }

    return device;
}

(2)CameraDeviceImpl

/frameworks/base/core/java/android/hardware/camera2/Impl/CameraDeviceImpl.java

Before continuing with the downward analysis to open the camera process, take a brief look at the setRemoteDevice method in the CameraDeviceImpl that was invoked, primarily to save the acquired remote device:

/**
 * Set remote device, which triggers initial onOpened/onUnconfigured callbacks
 *
 * <p>This function may post onDisconnected and throw CAMERA_DISCONNECTED if remoteDevice dies
 * during setup.</p>
 *
 */
public void setRemoteDevice(ICameraDeviceUser remoteDevice) throws CameraAccessException {
    synchronized(mInterfaceLock) {
        // TODO: Move from decorator to direct binder-mediated exceptions
        // If setRemoteFailure already called, do nothing
        if (mInError) return;

        mRemoteDevice = new ICameraDeviceUserWrapper(remoteDevice); //Add a layer of encapsulation to the remote device instance via ICameraDeviceUserWrapper.

        IBinder remoteDeviceBinder = remoteDevice.asBinder(); //Some basic settings for using the Binder mechanism.
        // For legacy camera device, remoteDevice is in the same process, and
        // asBinder returns NULL.
        if (remoteDeviceBinder != null) {
            try {
                remoteDeviceBinder.linkToDeath(this, /*flag*/ 0); //If the binder disappears, register a receiver for the flag information.
            } catch (RemoteException e) {
                CameraDeviceImpl.this.mDeviceHandler.post(mCallOnDisconnected);

                throw new CameraAccessException(CameraAccessException.CAMERA_DISCONNECTED,
                        "The camera device has encountered a serious error");
            }
        }

        mDeviceHandler.post(mCallOnOpened); //You need to trigger onOpened and onUnconfigured callbacks here, each of which is invoked by enabling a new thread through the mDeviceHandler.
        mDeviceHandler.post(mCallOnUnconfigured);
    }
}

(3) Runtime: Through the Binder mechanism, we call the connectDevice method remotely (called a function in C++, but it may be easier to say), which is implemented in the CameraService class.

(4)CameraService:

/frameworks/av/services/camera/libcameraservice/CameraService.cpp

Status CameraService::connectDevice(
        const sp<hardware::camera2::ICameraDeviceCallbacks>& cameraCb,
        const String16& cameraId,
        const String16& clientPackageName,
        int clientUid,
        /*out*/
        sp<hardware::camera2::ICameraDeviceUser>* device) {

    ATRACE_CALL();
    Status ret = Status::ok();
    String8 id = String8(cameraId);
    sp<CameraDeviceClient> client = nullptr;

　　 //The connectHelper method invoked here actually implements the connection logic (which is eventually invoked when HAL1).It is important to note that the set template types are ICameraDeviceCallbacks and CameraDeviceClient.
    ret = connectHelper<hardware::camera2::ICameraDeviceCallbacks,CameraDeviceClient>(cameraCb, id,
            CAMERA_HAL_API_VERSION_UNSPECIFIED, clientPackageName,
            clientUid, USE_CALLING_PID, API_2,
            /*legacyMode*/ false, /*shimUpdateOnly*/ false,
            /*out*/client);

    if(!ret.isOk()) {
        logRejected(id, getCallingPid(), String8(clientPackageName),
                ret.toString8());
        return ret;
    }

    *device = client; //client points to CameraDeviceClient, whose instance is the final return.
    return ret;
}

ConneHelper has a lot of content, ignoring the local analysis that we don't need to focus on yet:

template<class CALLBACK, class CLIENT>
Status CameraService::connectHelper(const sp<CALLBACK>& cameraCb, const String8& cameraId,
        int halVersion, const String16& clientPackageName, int clientUid, int clientPid,
        apiLevel effectiveApiLevel, bool legacyMode, bool shimUpdateOnly,
        /*out*/sp<CLIENT>& device) {
    binder::Status ret = binder::Status::ok();

    String8 clientName8(clientPackageName);

    /* Do something in */
    ......
    /* Do something out */

        sp<BasicClient> tmp = nullptr;
        //Call makeClient to generate a CameraDeviceClient instance.
        if(!(ret = makeClient(this, cameraCb, clientPackageName, cameraId, facing, clientPid,
                clientUid, getpid(), legacyMode, halVersion, deviceVersion, effectiveApiLevel,
                /*out*/&tmp)).isOk()) {
            return ret;
        }
        //Initialize the CLIENT instance.Note that the template type CLIENT here is the CameraDeviceClient, and the parameter mCameraProviderManager passed in is related to the HAL service. 
        client = static_cast<CLIENT*>(tmp.get());

        LOG_ALWAYS_FATAL_IF(client.get() == nullptr, "%s: CameraService in invalid state",
                __FUNCTION__);

        err = client->initialize(mCameraProviderManager);

    /* Do something in */
    ......
    /* Do something out */

    // Important: release the mutex here so the client can call back into the service from its
    // destructor (can be at the end of the call)
    device = client;
    return ret;
}

makeClient chooses to generate a specific Client instance based on the API version and the HAL version, and Client returns to the CameraDeviceImpl instance along the path analyzed earlier and is saved to the mRemoteDevice.

Status CameraService::makeClient(const sp<CameraService>& cameraService,
        const sp<IInterface>& cameraCb, const String16& packageName, const String8& cameraId,
        int facing, int clientPid, uid_t clientUid, int servicePid, bool legacyMode,
        int halVersion, int deviceVersion, apiLevel effectiveApiLevel,
        /*out*/sp<BasicClient>* client) {

    if (halVersion < 0 || halVersion == deviceVersion) {
        // Default path: HAL version is unspecified by caller, create CameraClient
        // based on device version reported by the HAL.
        switch(deviceVersion) {
          case CAMERA_DEVICE_API_VERSION_1_0:
            /* Do something in */
            ......
            /* Do something out */
          case CAMERA_DEVICE_API_VERSION_3_0:
          case CAMERA_DEVICE_API_VERSION_3_1:
          case CAMERA_DEVICE_API_VERSION_3_2:
          case CAMERA_DEVICE_API_VERSION_3_3:
          case CAMERA_DEVICE_API_VERSION_3_4:
            if (effectiveApiLevel == API_1) { // Camera1 API route
                sp<ICameraClient> tmp = static_cast<ICameraClient*>(cameraCb.get());
                *client = new Camera2Client(cameraService, tmp, packageName, cameraIdToInt(cameraId),
                        facing, clientPid, clientUid, servicePid, legacyMode);
            } else { // Camera2 API route: Instantiates the CameraDeviceClient class as a Client (note that the construction here passes in ICameraDeviceCallbacks, which is a remote callback to the CameraDeviceImpl)
                sp<hardware::camera2::ICameraDeviceCallbacks> tmp =
                        static_cast<hardware::camera2::ICameraDeviceCallbacks*>(cameraCb.get());
                *client = new CameraDeviceClient(cameraService, tmp, packageName, cameraId,
                        facing, clientPid, clientUid, servicePid);
            }
            break;
          default:
            // Should not be reachable
            ALOGE("Unknown camera device HAL version: %d", deviceVersion);
            return STATUS_ERROR_FMT(ERROR_INVALID_OPERATION,
                    "Camera device \"%s\" has unknown HAL version %d",
                    cameraId.string(), deviceVersion);
        }
    } else {
        /* Do something in */
        ......
        /* Do something out */
    }
    return Status::ok();
}

At this point, in the camera process, the call logic from App to CameraService is almost complete.

Summary of sketch:

Ps:

• CameraManager Globalis the real implementation layer, creating a connection to the CameraService of the JAVA layer to create a connection to the camera.
• CameraDeviceImpl is equivalent to the run context and replaces the JNI layer prior to Android N.

3. From CameraService to HAL Service

Because of the Treble mechanism included in Android O, CameraServer has a CameraService as its main body. It will look for an existing Provider service and add it to the internal CameraProvider Manager for management, all of which are done through remote calls.
The main body of one end of the Provider service is the CameraProvider, which is connected to the CameraHAL implementation layer of libhardware at its initialization and managed by the CameraModule.
After the process is started, the "carrier" of the connection is built (note that QCamera3HWI has not been created at this time), which can be simply represented by the following figure:

When the camera is turned on, a full connection to the layer is created, and the main calling logic is as follows:

Earlier, in CameraService::makeClient, a CameraDeviceClient was instantiated.Now let's start with its constructor and continue exploring the process of opening the camera.
The main activity of this section is at the Runtime layer, which is divided into CameraService and HAL Service sides for analysis.

(1)CameraDeviceClient : frameworks\av\services\camera\libcameraservice\api2\CameraDeviceClient.cpp

CameraDeviceClient::CameraDeviceClient(const sp<CameraService>& cameraService,
        const sp<hardware::camera2::ICameraDeviceCallbacks>& remoteCallback,
        const String16& clientPackageName,
        const String8& cameraId,
        int cameraFacing,
        int clientPid,
        uid_t clientUid,
        int servicePid) :
    Camera2ClientBase(cameraService, remoteCallback, clientPackageName,
                cameraId, cameraFacing, clientPid, clientUid, servicePid),  //Inherit its parent Camera2ClientBase 
    mInputStream(),
    mStreamingRequestId(REQUEST_ID_NONE),
    mRequestIdCounter(0),
    mPrivilegedClient(false) {

    char value[PROPERTY_VALUE_MAX];
    property_get("persist.camera.privapp.list", value, "");
    String16 packagelist(value);
    if (packagelist.contains(clientPackageName.string())) {
        mPrivilegedClient = true;
    }

    ATRACE_CALL();
    ALOGI("CameraDeviceClient %s: Opened", cameraId.string());
}

CameraService calls its initialization function after creating the CameraDeviceClient:

//Initialize function interface initialize that provides calls to the outside world.
status_t CameraDeviceClient::initialize(sp<CameraProviderManager> manager) {
    return initializeImpl(manager);
}

//The concrete implementation function of the initialization, where the template TProviderPtr is the CameraProviderManager class.
template<typename TProviderPtr>
//First initialize the parent class, notice that CameraProviderManager is passed in here.
status_t CameraDeviceClient::initializeImpl(TProviderPtr providerPtr) {
    ATRACE_CALL();
    status_t res;

    res = Camera2ClientBase::initialize(providerPtr);
    if (res != OK) {
        return res;
    }

    //Here is the creation and initialization of the FrameProcessor configuration, etc.
    String8 threadName;
    mFrameProcessor = new FrameProcessorBase(mDevice);
    threadName = String8::format("CDU-%s-FrameProc", mCameraIdStr.string());
    mFrameProcessor->run(threadName.string());

    mFrameProcessor->registerListener(FRAME_PROCESSOR_LISTENER_MIN_ID,
                                      FRAME_PROCESSOR_LISTENER_MAX_ID,
                                      /*listener*/this,
                                      /*sendPartials*/true);

    return OK;
}

(2)Camera2ClientBase: frameworks\av\services\camera\libcameraservice\common\Camera2ClientBase.cpp

template <typename TClientBase>　//Template TClientBase, specified as CameraDeviceClientBase when CameraDeviceClient inherits Camera2ClientBase.
Camera2ClientBase<TClientBase>::Camera2ClientBase( //The parameters associated with the construction, as well as the initialization list, note that TCamCallbacks are specified as ICameraDeviceCallbacks in the CameraDeviceClientBase. 
        const sp<CameraService>& cameraService,
        const sp<TCamCallbacks>& remoteCallback,
        const String16& clientPackageName,
        const String8& cameraId,
        int cameraFacing,
        int clientPid,
        uid_t clientUid,
        int servicePid):
        TClientBase(cameraService, remoteCallback, clientPackageName,
                cameraId, cameraFacing, clientPid, clientUid, servicePid),
        mSharedCameraCallbacks(remoteCallback),
        mDeviceVersion(cameraService->getDeviceVersion(TClientBase::mCameraIdStr)),
        mDeviceActive(false)
{
    ALOGI("Camera %s: Opened. Client: %s (PID %d, UID %d)", cameraId.string(),
            String8(clientPackageName).string(), clientPid, clientUid);

    mInitialClientPid = clientPid;
    mDevice = new Camera3Device(cameraId); //Created a Camera3Device.
    LOG_ALWAYS_FATAL_IF(mDevice == 0, "Device should never be NULL here.");
}

Go back and look at the initialization function:

template <typename TClientBase> //Initialize the function interface, the real implementation is in initializeImpl.
status_t Camera2ClientBase<TClientBase>::initialize(sp<CameraProviderManager> manager) {
    return initializeImpl(manager);
}

//TClientBase corresponds to CameraDeviceClientBase, while TProviderPtr corresponds to CameraProviderManager.
template <typename TClientBase>
template <typename TProviderPtr>
status_t Camera2ClientBase<TClientBase>::initializeImpl(TProviderPtr providerPtr) {
    ATRACE_CALL();
    ALOGV("%s: Initializing client for camera %s", __FUNCTION__,
          TClientBase::mCameraIdStr.string());
    status_t res;

    // Verify ops permissions
    res = TClientBase::startCameraOps(); //Call the startCameraOps method of CameraDeviceClientBase to check the permissions of the ops.
    if (res != OK) {
        return res;
    }

    if (mDevice == NULL) {
        ALOGE("%s: Camera %s: No device connected",
                __FUNCTION__, TClientBase::mCameraIdStr.string());
        return NO_INIT;
    }

    res = mDevice->initialize(providerPtr); //Initialize an instance of Camera3Device, noting that CameraProviderManager is passed in here.
    if (res != OK) {
        ALOGE("%s: Camera %s: unable to initialize device: %s (%d)",
                __FUNCTION__, TClientBase::mCameraIdStr.string(), strerror(-res), res);
        return res;
    }

　　 //Set the Notify callback in the Camera3Device instance.
    wp<CameraDeviceBase::NotificationListener> weakThis(this);
    res = mDevice->setNotifyCallback(weakThis);

    return OK;
}

(3)Camera3Device:frameworks\av\services\camera\libcameraservice\device3\Camera3Device.cpp

Camera3Device::Camera3Device(const String8 &id):
        mId(id),
        mOperatingMode(NO_MODE),
        mIsConstrainedHighSpeedConfiguration(false),
        mStatus(STATUS_UNINITIALIZED),
        mStatusWaiters(0),
        mUsePartialResult(false),
        mNumPartialResults(1),
        mTimestampOffset(0),
        mNextResultFrameNumber(0),
        mNextReprocessResultFrameNumber(0),
        mNextShutterFrameNumber(0),
        mNextReprocessShutterFrameNumber(0),
        mListener(NULL),
        mVendorTagId(CAMERA_METADATA_INVALID_VENDOR_ID)
{
    ATRACE_CALL();
　　//Two callback interfaces are set in this observation constructor:
    camera3_callback_ops::notify = &sNotify;
    camera3_callback_ops::process_capture_result = &sProcessCaptureResult;
    ALOGV("%s: Created device for camera %s", __FUNCTION__, mId.string());
}

It has a long initialization function, omitting the operation on RequestMetadataQueue.

status_t Camera3Device::initialize(sp<CameraProviderManager> manager) {
    ATRACE_CALL();
    Mutex::Autolock il(mInterfaceLock);
    Mutex::Autolock l(mLock);

    ALOGV("%s: Initializing HIDL device for camera %s", __FUNCTION__, mId.string());
    if (mStatus != STATUS_UNINITIALIZED) {
        CLOGE("Already initialized!");
        return INVALID_OPERATION;
    }
    if (manager == nullptr) return INVALID_OPERATION;

    sp<ICameraDeviceSession> session;
    ATRACE_BEGIN("CameraHal::openSession");
    status_t res = manager->openSession(mId.string(), this, //openSession method of CameraProviderManager is called to open remote Session
            /*out*/ &session);
    ATRACE_END();
    if (res != OK) {
        SET_ERR_L("Could not open camera session: %s (%d)", strerror(-res), res);
        return res;
    }

    /* Do something in */
    ......
    /* Do something out */

    return initializeCommonLocked();
}

(4)CameraProviderManager: frameworks\av\services\camera\libcameraservice\common\CameraProviderManager.cpp

status_t CameraProviderManager::openSession(const std::string &id,
        const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback,
        /*out*/
        sp<hardware::camera::device::V3_2::ICameraDeviceSession> *session) {

    std::lock_guard<std::mutex> lock(mInterfaceMutex);

    auto deviceInfo = findDeviceInfoLocked(id, //First call findDeviceInfoLocked to get the HAL3-related DeviceInfo3
            /*minVersion*/ {3,0}, /*maxVersion*/ {4,0});
    if (deviceInfo == nullptr) return NAME_NOT_FOUND;

    auto *deviceInfo3 = static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo);

    Status status;
    hardware::Return<void> ret;
　　 //Creates a CameraDeviceSession instance by remotely calling the open method of CameraDevice and returns its local calling interface by entering a session.
    ret = deviceInfo3->mInterface->open(callback, [&status, &session]
            (Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) {
                status = s;
                if (status == Status::OK) {
                    *session = cameraSession;
                }
            });
    if (!ret.isOk()) {
        ALOGE("%s: Transaction error opening a session for camera device %s: %s",
                __FUNCTION__, id.c_str(), ret.description().c_str());
        return DEAD_OBJECT;
    }
    return mapToStatusT(status);
}

(5)CameraDevice:hardware\interfaces\camera\device\3.2\default\CameraDevice.cpp

An instance of CameraDevice actually exists after the HAL Service has been initialized.Previously, through the deviceInfo interface in CameraProviderManager, you invoked the open method of a remote CameraDevice instance, so let's look at its code implementation:

Return<void> CameraDevice::open(const sp<ICameraDeviceCallback>& callback, open_cb _hidl_cb)  {
    Status status = initStatus();
    sp<CameraDeviceSession> session = nullptr;

    if (callback == nullptr) {
        ALOGE("%s: cannot open camera %s. callback is null!",
                __FUNCTION__, mCameraId.c_str());
        _hidl_cb(Status::ILLEGAL_ARGUMENT, nullptr);
        return Void();
    }

    if (status != Status::OK) {
        /* Do something in */
        ......
        /* Do something out */
    } else {
        mLock.lock();

        /* Do something in */
        ......
        /* Do something out */

        /** Open HAL device */
        status_t res;
        camera3_device_t *device;

        ATRACE_BEGIN("camera3->open");
        res = mModule->open(mCameraId.c_str(), //Note that mModule is configured when the HAL Service is initialized, and it encapsulates the Amera HAL interface loaded from the libhardware library one step down to the QCamera3HWI construction process.
                reinterpret_cast<hw_device_t**>(&device));
        ATRACE_END();

        /* Do something in */
        ......
        /* Do something out */

　　　　 //Create a session and hold it by an internal member, mSession, which implements the function creatSession.
        session = createSession(
                device, info.static_camera_characteristics, callback);

        /* Do something in */
        ......
        /* Do something out */

        mSession = session;

        IF_ALOGV() {
            session->getInterface()->interfaceChain([](
                ::android::hardware::hidl_vec<::android::hardware::hidl_string> interfaceChain) {
                    ALOGV("Session interface chain:");
                    for (auto iface : interfaceChain) {
                        ALOGV("  %s", iface.c_str());
                    }
                });
        }
        mLock.unlock();
    }
    _hidl_cb(status, session->getInterface());
    return Void();
}

A CameraDeviceSession is created directly in creatSession.Of course, the internal initialization function is called in its constructor, and then it enters the initialization process of the HAL interface layer QCamera3HWI, so the process of opening cameras from CameraService to HAL Service is almost complete.
Summary of sketch:

4. From HAL Service to Camera HAL

In HAL3, the interface transformation layer (and flow resolution layer) of Camera HAL is assumed by QCamera3HardwareInterface, while the interface layer and implementation layer are basically the same as those in HAL1, both in mm_camera_interface.c and mm_camera.c.
When and how did an instance of the interface transformation layer be created and initialized, and how did it interact with the interface layer and the implementation layer when it was created?The main calling process is shown in the following figure:
　　

(1)CameraModule(HAL Servic) : hardware\interfaces\camera\common\1.0\default\CameraModule.cpp

Last time, in the implementation of CameraDevice::open, mModule->open, or CameraModule::open, is called. From the code point of view, it doesn't do much, mainly calling mModule->common.methods->open to go to the next level of process.
(Note here that open is a function pointer that points to the camera_device_open method of QCamera2Factory. As to why it is related to QCamera2Factory, look back at the HAL Service startup initialization process.

int CameraModule::open(const char* id, struct hw_device_t** device) {
    int res;
    ATRACE_BEGIN("camera_module->open");
    res = filterOpenErrorCode(mModule->common.methods->open(&mModule->common, id, device));
    ATRACE_END();
    return res;
}

(2)QCamera2Factory(Camera HAL): hardware\qcom\camera\qcamera2\QCamera2Factory.cpp

/*===========================================================================
 * FUNCTION   : camera_device_open
 *
 * DESCRIPTION: static function to open a camera device by its ID
 *
 * PARAMETERS :
 *   @camera_id : camera ID
 *   @hw_device : ptr to struct storing camera hardware device info
 *
 * RETURN     : int32_t type of status
 *              NO_ERROR  -- success
 *              none-zero failure code
 *==========================================================================*/
int QCamera2Factory::camera_device_open(
    const struct hw_module_t *module, const char *id,
    struct hw_device_t **hw_device)
{
    /* Do something in */
    ......
    /* Do something out */

#ifdef QCAMERA_HAL1_SUPPORT //Notice here that compatible operations for HAL1 have been added through macro definitions.You are actually calling cameraDeviceOpen for the next step.
    if(gQCameraMuxer)
        rc =  gQCameraMuxer->camera_device_open(module, id, hw_device);
    else
#endif
        rc = gQCamera2Factory->cameraDeviceOpen(atoi(id), hw_device);
    return rc;
}

struct hw_module_methods_t QCamera2Factory::mModuleMethods = {
    .open = QCamera2Factory::camera_device_open, //This specifies the open function pointer mentioned earlier as the camera_device_open method.
};

Work of cameraDeviceOpen:

/*===========================================================================
 * FUNCTION   : cameraDeviceOpen
 *
 * DESCRIPTION: open a camera device with its ID
 *
 * PARAMETERS :
 *   @camera_id : camera ID
 *   @hw_device : ptr to struct storing camera hardware device info
 *
 * RETURN     : int32_t type of status
 *              NO_ERROR  -- success
 *              none-zero failure code
 *==========================================================================*/
int QCamera2Factory::cameraDeviceOpen(int camera_id,
                    struct hw_device_t **hw_device)
{
    /* Do something in */
    ......
    /* Do something out */

    if ( mHalDescriptors[camera_id].device_version == CAMERA_DEVICE_API_VERSION_3_0 ) {
        QCamera3HardwareInterface *hw = new QCamera3HardwareInterface(mHalDescriptors[camera_id].cameraId, //First, an instance of QCamera3HardwareInterface is created.
                mCallbacks);
        if (!hw) {
            LOGE("Allocation of hardware interface failed");
            return NO_MEMORY;
        }
        rc = hw->openCamera(hw_device); //Call the openCamera method of the instance.
        if (rc != 0) {
            delete hw;
        }
    }

    /* Do something in */
    ......
    /* Do something out */

    return rc;
}

(3)QCamera3HardwareInterface : hardware\qcom\camera\qcamera2\hal3\QCamera3HWI.cpp

The first thing to note is the definition of the internal member mCameraOps.When constructing an instance, there is mCameraDevice.ops = &mCameraOps; (Key Point)

camera3_device_ops_t QCamera3HardwareInterface::mCameraOps = {
    .initialize                         = QCamera3HardwareInterface::initialize,
    .configure_streams                  = QCamera3HardwareInterface::configure_streams,
    .register_stream_buffers            = NULL,
    .construct_default_request_settings = QCamera3HardwareInterface::construct_default_request_settings,
    .process_capture_request            = QCamera3HardwareInterface::process_capture_request,
    .get_metadata_vendor_tag_ops        = NULL,
    .dump                               = QCamera3HardwareInterface::dump,
    .flush                              = QCamera3HardwareInterface::flush,
    .reserved                           = {0},
};

Let's move on to the openCamera implementation:

int QCamera3HardwareInterface::openCamera(struct hw_device_t **hw_device)
{
    /* Do something in */
    ......
    /* Do something out */

    rc = openCamera(); //Call another openCamera method, which is part of the implementation.
    if (rc == 0) {
        *hw_device = &mCameraDevice.common; //When the camera is turned on successfully, the common part of the device structure is returned through the double pointer hw_device.
    } else
        *hw_device = NULL;

    /* Do something in */
    ......
    /* Do something out */
    return rc;
}

int QCamera3HardwareInterface::openCamera()
{
    /* Do something in */
    ......
    /* Do something out */

    rc = camera_open((uint8_t)mCameraId, &mCameraHandle);  //This is where you begin to enter the interface layer, which calls the camera_open interface in the interface layer.Note that mCameraHandle is obtained here.

    /* Do something in */
    ......
    /* Do something out */

    rc = mCameraHandle->ops->register_event_notify(mCameraHandle->camera_handle, //Notice that a camEvtHandle is passed in here
            camEvtHandle, (void *)this);

    /* Do something in */
    ......
    /* Do something out */ 

    rc = mCameraHandle->ops->get_session_id(mCameraHandle->camera_handle, 
        &sessionId[mCameraId]);

    /* Do something in */
    ......
    /* Do something out */

    return NO_ERROR;
}

Above is the section on openCamera in the Interface Transformation Layer, and below you'll look at its initialization function.As previously analyzed, when a CameraDeviceSession instance is created, its internal initialization method is called, which contains the initialize method that calls QCamera3HWI

int QCamera3HardwareInterface::initialize(const struct camera3_device *device,
                                  const camera3_callback_ops_t *callback_ops)
{
    LOGD("E");
    QCamera3HardwareInterface *hw =
        reinterpret_cast<QCamera3HardwareInterface *>(device->priv);
    if (!hw) {
        LOGE("NULL camera device");
        return -ENODEV;
    }

    int rc = hw->initialize(callback_ops); //A function that calls the real implementation of the initialization logic
    LOGD("X");
    return rc;
}

int QCamera3HardwareInterface::initialize(
        const struct camera3_callback_ops *callback_ops)
{
    ATRACE_CALL();
    int rc;

    LOGI("E :mCameraId = %d mState = %d", mCameraId, mState);
    pthread_mutex_lock(&mMutex);

    // Validate current state
    switch (mState) {
        case OPENED:
            /* valid state */
            break;
        default:
            LOGE("Invalid state %d", mState);
            rc = -ENODEV;
            goto err1;
    }

    rc = initParameters(); //Parameters (mParameters) are initialized, note that the parameters and CameraParameter s here are different and are the structure of the metadata_buffer related parameters.
    if (rc < 0) {
        LOGE("initParamters failed %d", rc);
        goto err1;
    }
    mCallbackOps = callback_ops; //Here you associate camera3_call_back_ops with mCallbackOps.

    mChannelHandle = mCameraHandle->ops->add_channel( //Get the handle mChannelHandle by calling the mm_camera_intf_add_channel in mm_camera_interface.c.
            mCameraHandle->camera_handle, NULL, NULL, this);
    if (mChannelHandle == 0) {
        LOGE("add_channel failed");
        rc = -ENOMEM;
        pthread_mutex_unlock(&mMutex);
        return rc;
    }

    pthread_mutex_unlock(&mMutex);
    mCameraInitialized = true;
    mState = INITIALIZED;
    LOGI("X");
    return 0;

err1:
    pthread_mutex_unlock(&mMutex);
    return rc;
}

(4) mm_camera_interface.c (interface layer):

hardware\qcom\camera\qcamera2\stack\mm-camera-interface\src\mm_camera_interface.c

Cam_open does not do much, omitting the memory allocation and initialization for cam_obj.In fact, the mm_camera_open in the implementation layer is called to actually open the camera device, and the various information of the device is populated in the cam_obj structure.

int32_t camera_open(uint8_t camera_idx, mm_camera_vtbl_t **camera_vtbl)
{
    int32_t rc = 0;
    mm_camera_obj_t *cam_obj = NULL;

    /* Do something in */
    ......
    /* Do something out */

    rc = mm_camera_open(cam_obj);

    /* Do something in */
    ......
    /* Do something out */
}

The mm_camera_intf_add_channel code for initialization calls is as follows:

static uint32_t mm_camera_intf_add_channel(uint32_t camera_handle,
                                           mm_camera_channel_attr_t *attr,
                                           mm_camera_buf_notify_t channel_cb,
                                           void *userdata)
{
    uint32_t ch_id = 0;
    mm_camera_obj_t * my_obj = NULL;

    LOGD("E camera_handler = %d", camera_handle);
    pthread_mutex_lock(&g_intf_lock);
    my_obj = mm_camera_util_get_camera_by_handler(camera_handle);

    if(my_obj) {
        pthread_mutex_lock(&my_obj->cam_lock);
        pthread_mutex_unlock(&g_intf_lock);
        ch_id = mm_camera_add_channel(my_obj, attr, channel_cb, userdata); //Get a channel id, which is its handle, by calling the mm_camera_add_channel of the implementation layer.
    } else {
        pthread_mutex_unlock(&g_intf_lock);
    }
    LOGD("X ch_id = %d", ch_id);
    return ch_id;
}

(5) mm_camera.c (implementation layer):

hardware\qcom\camera\qcamera2\stack\mm-camera-interface\src\mm_camera.c

Finally, to the bottom level of the implementation, mm_camera_open's main job is to fill my_obj and start and initialize some thread-related things, which I omitted here.

int32_t mm_camera_open(mm_camera_obj_t *my_obj)
{
    char dev_name[MM_CAMERA_DEV_NAME_LEN];
    int32_t rc = 0;
    int8_t n_try=MM_CAMERA_DEV_OPEN_TRIES;
    uint8_t sleep_msec=MM_CAMERA_DEV_OPEN_RETRY_SLEEP;
    int cam_idx = 0;
    const char *dev_name_value = NULL;
    int l_errno = 0;
    pthread_condattr_t cond_attr;

    LOGD("begin\n");

    if (NULL == my_obj) {
        goto on_error;
    }
    dev_name_value = mm_camera_util_get_dev_name(my_obj->my_hdl); //The function called here is to get a handle to my_obj, which is not analyzed in depth here.
    if (NULL == dev_name_value) {
        goto on_error;
    }
    snprintf(dev_name, sizeof(dev_name), "/dev/%s",
             dev_name_value);
    sscanf(dev_name, "/dev/video%d", &cam_idx);
    LOGD("dev name = %s, cam_idx = %d", dev_name, cam_idx);

    do{
        n_try--;
        errno = 0;
        my_obj->ctrl_fd = open(dev_name, O_RDWR | O_NONBLOCK); //Read the file descriptor of the device file and save it in my_obj->ctrl_fd.
        l_errno = errno;
        LOGD("ctrl_fd = %d, errno == %d", my_obj->ctrl_fd, l_errno);
        if((my_obj->ctrl_fd >= 0) || (errno != EIO && errno != ETIMEDOUT) || (n_try <= 0 )) {
            break;
        }
        LOGE("Failed with %s error, retrying after %d milli-seconds",
              strerror(errno), sleep_msec);
        usleep(sleep_msec * 1000U);
    }while (n_try > 0);

    if (my_obj->ctrl_fd < 0) {
        LOGE("cannot open control fd of '%s' (%s)\n",
                  dev_name, strerror(l_errno));
        if (l_errno == EBUSY)
            rc = -EUSERS;
        else
            rc = -1;
        goto on_error;
    } else {
        mm_camera_get_session_id(my_obj, &my_obj->sessionid); //Once you have successfully obtained the file descriptor, you need to get the session id.
        LOGH("Camera Opened id = %d sessionid = %d", cam_idx, my_obj->sessionid);
    }

    /* Do something in */
    ......
    /* Do something out */

    /* unlock cam_lock, we need release global intf_lock in camera_open(),
     * in order not block operation of other Camera in dual camera use case.*/
    pthread_mutex_unlock(&my_obj->cam_lock);
    return rc;
}

For the relevant parts of the initialization, the mm_camera_add_channel code is as follows:

uint32_t mm_camera_add_channel(mm_camera_obj_t *my_obj,
                               mm_camera_channel_attr_t *attr,
                               mm_camera_buf_notify_t channel_cb,
                               void *userdata)
{
    mm_channel_t *ch_obj = NULL;
    uint8_t ch_idx = 0;
    uint32_t ch_hdl = 0;

　　 //Find the first state NOTUSED from the existing Channel, get it in ch_obj
    for(ch_idx = 0; ch_idx < MM_CAMERA_CHANNEL_MAX; ch_idx++) {
        if (MM_CHANNEL_STATE_NOTUSED == my_obj->ch[ch_idx].state) {
            ch_obj = &my_obj->ch[ch_idx];
            break;
        }
    }

　　/*Initialize the ch_obj structure.First call mm_camera_util_generate_handler to generate a handle for it (which is also the return value of the function).
　　 *Then set the state to STOPPED, note that my_obj's pointer and its session id are also saved here, and finally mm_channel_init is called to initialize Channel.*/
    if (NULL != ch_obj) {
        /* initialize channel obj */
        memset(ch_obj, 0, sizeof(mm_channel_t));
        ch_hdl = mm_camera_util_generate_handler(ch_idx);
        ch_obj->my_hdl = ch_hdl;
        ch_obj->state = MM_CHANNEL_STATE_STOPPED;
        ch_obj->cam_obj = my_obj;
        pthread_mutex_init(&ch_obj->ch_lock, NULL);
        ch_obj->sessionid = my_obj->sessionid;
        mm_channel_init(ch_obj, attr, channel_cb, userdata);
    }

    pthread_mutex_unlock(&my_obj->cam_lock);

    return ch_hdl;
}

Summary of sketch:

All in all, after this operation, the entire connection from top to bottom of the camera has been made, and the next step should be to get preview data as long as APP sends Preview's Request following the process.

3. Core concepts: Request

Request is the most important concept throughout the camera2 data processing process. The application framework is to get the desired result by sending a request to the camera2 subsystem.
request has the following important features:

• A request can correspond to a series of result s.
• The request should contain all the necessary configuration information and be stored in metadata.For example: resolution and pixel format; control information of sensor, lens, flash, etc; 3A operation mode; RAW to YUV processing control; and generation of statistical information.
• request needs to carry the corresponding surface (stream inside the frame) to receive the returned image.
• Multiple requests can be in-flight at the same time, and submit request s are non-blocking.That is, you can submit a new request if the last request has not been processed.
• Queue request s are always handled as FIFO s.
• The snapshot request has a higher priority than the preview request.

1. The overall process of request is as follows:

• open process (black arrow lines)
  CameraManager registers AvailabilityCallback callbacks to receive notifications of changes in the availability status of camera devices.
  CameraManager obtains the currently available camera id by calling getCameraIdList(), and obtains the characteristics of the specified camera device through the getCameraCharacteristcs() function.
  CameraManager calls openCamera() to open the specified camera device and returns a CameraDevice object through which to manipulate the specific camera device.
  Create a session using the createCaptureSession() of the CameraDevice object, and data requests (preview, take photos, and so on) are made through sessions.When creating a session, you need to provide Surface as a parameter to receive the returned image.

• configure stream process (blue arrow line)
  Request Surface, such as the OUTPUT STREAMS DESTINATIONS box in the figure above, to receive the image returned by session as a parameter when it is created.
  Once session s are created, surface is configured as a stream for the framework.In the framework, stream defines the size and format of the image.
  Each request needs to carry a target surface that specifies to which stream the returned image belongs to the configure.

• request process (orange arrow line)
  The CameraDevice object creates a request through createCaptureRequest(), and each reqeust requires surface s and settings (settings are metadata, and all configuration information contained in the request is placed in metadata).
  Requests can be sent to the framework using APIs such as session capture(), captureBurst(), setStreamingRequest(), setStreamingBurst().
  The previewed request is sent through setStreamingRequest(), setStreamingBurst(), and is called only once.Set the request into the repeating request list.As long as there is no request in the pending request queue, copy the request from the repeating list into the pending queue.
  The request to take a photo is sent by capture(), captureBurst(), and is called every time a photo needs to be taken.Each trigger puts the request directly into the pending request queue, so requests taken take a photo have a higher priority than requests previewed.
  The in-progress queue represents the queue of the request currently being processed, and each time it is processed, a new request is pulled out of the pending queue and placed here.

• Data return process (purple arrow line)
  The data returned at the hardware level is returned in the result, and the response is called back by session's capture callback.

2. How requests are handled in HAL

(1) framework sends asynchronous request s to hal.
(2) hal must process requests sequentially, returning timestamp (shutter, the time at which the frame was generated), metadata, image buffers for each request.
(3) For each type of steam referenced by request, the result must be returned as FIFO.For example, for a preview stream, result id 9 must be returned before result id 10.However, the stream taken can currently only be returned to result id 7 because the stream used for photographing and previewing is different.
(4) All the information hal needs is received by the metadata that request carries, and all the information hal needs to return is returned by the metadata that result carries.

The overall flow of HAL processing request s is shown below.

• request process (black arrow line)
  framework submits it request asynchronously to hal, Hal processes it in turn, and returns the result.
  Each request submit ted to hal must carry a stream.Streams are divided into input stream and output stream:input stream whose corresponding buffers are buffers of existing image data, hal reprocess these buffers; output stream whose corresponding buffers are empty buffer s, hal fills these buffers with the generated image data.

• input stream process (INPUT STREAM 1 for images)
  request carries input stream and input buffer to hal.
  hal reprocess, then the new image data is repopulated into the buffer and returned to the framework.

• output stream processing flow (OUTPUT STREAM 1 for images...N)
  request carries output stream and output buffer to hal.
  hal writes the image data to the buffer and returns it to the frameowork after a series of modules.

Personal blog: https://www.letcos.top/