I preface

The core design idea of Linux containerexecutor is to give the NodeManager initiator root permission, so that it has sufficient permission to perform some operations as any user, so that the NodeManager executor can modify the owner of the directories and files used by the Container to the application submitter, and take the application submitter as its own
Run containers in duplicate to prevent various security risks caused by all containers running as NodeManager executors. For example, prevent users from executing commands in the Container that only NodeManager users have permission to execute (commands to kill other applications, close or kill NodeManager processes, etc.).
In order to realize the above mechanism, NodeManager implements a tool with setuid function Container executor in C language. It has root permission and can complete any operations, such as creating Cgroups hierarchical tree, setting Cgroups attributes, etc. By calling this executable file, Linux containerexecutor can modify some properties of the Container to limit illegal operations of the Container (such as closing NodeManager, killing NodeManager, etc.).

II to configure

Suppose that it is grouped as hadoop,

2.1. yarn-site.xml configuration

    <!--    Set use LinuxContainerExecutor-->
    <property>
        <name>yarn.nodemanager.container-executor.class</name>
        <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
    </property>

    <!--    set up container-executor Location-->
    <property>
        <name>yarn.nodemanager.linux-container-executor.path</name>
        <value>/opt/tools/hadoop-3.2.1/bin/container-executor</value>
    </property>

    <!-- NM of Unix User group, need to follow container-executor.cfg The configuration inside is consistent, which is mainly used to verify whether there is secure access container-executor Binary permissions -->
    <property>
        <name>yarn.nodemanager.linux-container-executor.group</name>
        <value>hadoop</value>
    </property>

    <!-- Cluster in nonsecure Is there a limit when the mode is container Starting user of, true: container Use unified user startup false: container Start with task user -->
    <property>
        <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users</name>
        <value>true</value>
    </property>

    <!-- Cluster in nonsecure Mode and on container When user restriction is enabled, the user is used uniformly. If it is not set, it defaults to nobody -->
    <property>
        <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name>
        <value>yarn,henghe,root</value>
    </property>

2.2. container-executor.cfg configuration

container-executor.cfg is the configuration file of the container executor binary program, and the check will be read when it starts.
The specific attributes are as follows:

• yarn. nodemanager. linux-container-executor. Group: the Unix user group of nm, which needs to be connected with yarn site Consistent in XML.
• allowed.system.users: allowed system users. Multiple users are separated by ','. It can be left unset, that is, all users are allowed.
• banned.users: prohibited users. Multiple users are separated by ','.
• min.user.id: the minimum uid of the allowed user to prevent other super users.

yarn.nodemanager.linux-container-executor.group=hadoop
allowed.system.users=hdfs,yarn,henghe
banned.users=root,bin
min.user.id=1

2.3. Setting container executor permissions

The owner of the container executor binary must be root, and the generic group must be the same as the NM generic group (hadoop). At the same time, its permission must be set to 6050 to give it the permission of setuid to start the container with different users.

chown root:hadoop /usr/lib/hadoop-yarn/bin/container-executor 
chmod 6050 /usr/lib/hadoop-yarn/bin/container-executor

2.4. Set container executor CFG permissions

container-executor. The CFG binary owner must be root, and the membership group must be the same as the NM membership group (hadoop). At the same time, its permission must be set to 0400 to ensure that it is read-only and not writable

chown root:hadoop /etc/hadoop/conf/container-executor.cfg
chmod 0400 /etc/hadoop/conf/container-executor.cfg

2.5. Set the yarn local and yarn log dirs permissions

All configured yarn local and yarn log directories must be owned by yarn:hadoop. This general cluster has been modified when it is built and does not need to be processed.

2.6. Task user management

When security authentication is not enabled in the cluster, that is, the user verification method of the cluster is Hadoop security. When Authentication = simple, if you enable the user limit yarn nodemanager. linux-container-executor. nonsecure-mode. Limit users = false, that is, when using the task submitting user to run its container, you need to add the required users to the machine one by one through the useradd command on all NM nodes of the cluster in advance, otherwise the task will fail because the specified user cannot be found. When the cluster is large and there are many users, adding users is cumbersome and difficult to maintain. Therefore, based on our practical experience, it is recommended to open the user limit and uniformly use the yarn user to start the container.

However, when the cluster is in the security mode, kerberos is basically enabled for permission authentication. Only the task submitting user is allowed to start the container. At this time, it is hard to add all the users in the yarn cluster to the NM node. However, in order not to be so troublesome, I modified this logic to allow unified yarn users to perform user tasks under security. For the specific implementation method, you can see that yarn uses a unified user to run the container in the safe mode.

When the above operations are completed, restart NM and LCE will be enabled.

2.7. matters needing attention

Container executor and container executor The permissions of the cfg file must be modified as required, otherwise NM will fail to start. If NM still reports a permission problem after changing the permission, it is also necessary to change the user group of the directory where the file is located to root:hadoop and the permission to 755.

• Permission and ownership allocation table involving directory

Filesystem	Path	User:Group	Permissions
local	container-executor	root:hadoop	—Sr-s—
local	etc/hadoop/container-executor.cfg	root:hadoop	r-------
local	yarn.nodemanager.local-dirs	yarn:hadoop	drwxr-xr-x
local	yarn.nodemanager.log-dirs	yarn:hadoop	drwxr-xr-x

2.8. Reference link:

https://smarthanwang.github.io/2019/10/12/yarn-container-executor/
https://zhuanlan.zhihu.com/p/38552508
https://makeling.github.io/bigdata/dcb921f7.html

III Construction method

There are two construction methods. The difference is whether the input parameter is passed into the execution engine of Linux container runtime type

  /**
   * Default constructor to allow for creation through reflection.
   */
  public LinuxContainerExecutor() {
  }

  /**
   * Create a LinuxContainerExecutor with a provided
   * {@link LinuxContainerRuntime}.  Used primarily for testing.
   *
   * @param linuxContainerRuntime the runtime to use
   */
  public LinuxContainerExecutor(LinuxContainerRuntime linuxContainerRuntime) {
    this.linuxContainerRuntime = linuxContainerRuntime;
  }

The initialization entry is the NodeManager#serviceInt method Initialize through the createContainerExecutor method

  // According to yarn nodemanager. container-executor. class : org. apache. hadoop. yarn. server. nodemanager. LinuxContainerExecutor
  @VisibleForTesting
  protected ContainerExecutor createContainerExecutor(Configuration conf) {
  	// Build instances through reflection 
    return ReflectionUtils.newInstance(
        conf.getClass(YarnConfiguration.NM_CONTAINER_EXECUTOR,
            DefaultContainerExecutor.class, ContainerExecutor.class), conf);
  }

IV init

Load yarn nodemanager. container-executor. Org. Configured in the class parameter apache. hadoop. yarn. server. nodemanager. LinuxContainerExecutor .
Generate a LinuxContainerExecutor instance through reflection The init method is then initialized

ps: because the mac computer can't execute, I commented out the code to check whether the container executor is available Self verification required
Verified shell instruction: container executor -- checksetup

  @Override
  public void init(Context context) throws IOException {
    Configuration conf = super.getConf();
    this.nmContext = context;

    // Send command to executor which will just start up,
    // verify configuration/permissions and exit
    try {
      PrivilegedOperation checkSetupOp = new PrivilegedOperation(
          PrivilegedOperation.OperationType.CHECK_SETUP);
      PrivilegedOperationExecutor privilegedOperationExecutor =
          getPrivilegedOperationExecutor();

      // The mac environment will definitely report an error. In order to DEBUG, it is commented out
      // privilegedOperationExecutor.executePrivilegedOperation(checkSetupOp,false);
    } catch (Exception e) {
      //   todo commented out validation for DEBUG
//    int exitCode = e.getExitCode();
//    LOG.warn("Exit code from container executor initialization is : " + exitCode, e);
//    throw new IOException("Linux container executor not configured properly"   + " (error=" + exitCode + ")", e);
    }

    try {
      resourceHandlerChain = ResourceHandlerModule
          .getConfiguredResourceHandlerChain(conf, nmContext);
      if (LOG.isDebugEnabled()) {
        final boolean enabled = resourceHandlerChain != null;
        LOG.debug("Resource handler chain enabled = " + enabled);
      }
      if (resourceHandlerChain != null) {
        LOG.debug("Bootstrapping resource handler chain: " +
            resourceHandlerChain);
        resourceHandlerChain.bootstrap(conf);
      }
    } catch (ResourceHandlerException e) {
      LOG.error("Failed to bootstrap configured resource subsystems! ", e);
      throw new IOException(
          "Failed to bootstrap configured resource subsystems!");
    }

    try {
      if (linuxContainerRuntime == null) {
        LinuxContainerRuntime runtime = new DelegatingLinuxContainerRuntime();

        runtime.initialize(conf, nmContext);
        this.linuxContainerRuntime = runtime;
      }
    } catch (ContainerExecutionException e) {
      LOG.error("Failed to initialize linux container runtime(s)!", e);
      throw new IOException("Failed to initialize linux container runtime(s)!");
    }

    resourcesHandler.init(this);
  }

Here we can see that the default LinuxContainerRuntime engine used is DelegatingLinuxContainerRuntime

V ContainerExecutor flowchart

Review the execution process of the overall ContainerExecutor. Here, we are most concerned about the prepareContainer and launchContainer methods

5.1. prepareContainer

5.2. launchContainer

Because I can't debug, I'll come here first and watch it when I'm free later