Author | sparrow
Source| Alibaba cloud official account

This article is from Arthas. It was submitted in March 2021. The way of participation in the award-winning essay in April can be seen at the end of the article.

The project initially used Arthas for two purposes:

1. Through arthas, we can solve the problem of realizing the performance problem analysis tool of test environment, performance test environment and production environment.
2. By using the combination of jad, mc and redefine functions, the ability of hot update of some node codes in the production environment is realized.

Technology selection related

Because the company has not yet established a relatively unified production micro service configuration and state management capability, the R & D and operation and maintenance of their respective systems are relatively independent. Now the project uses the framework structure of Spring Cloud and Eureka, which matches the basic support ability of SBA. At the same time, SBA can provide service awareness, log level configuration management, and many management plug-ins based on actor JVM and Spring container, which can meet the needs of basic use.

During the investigation, the overall version of Arthas is 3.4.5, which provides the tunnel server mode based on Webconsole. Through the previous link article, it has been practiced and can be integrated with SBA. Because the project itself has no historical burden, SBA version 2.0 is adopted in the actual integration process to provide more management functions and graphical interface capabilities. Other advantages:

• The web console interface is embedded with SBA overall password login and web page permission management, so that the functions of relevant arthas web console can be used only after logging in SBA.

• Based on the jmx management of jolokia core open target service process that SBA client relies on, the jmx interface reuses the relevant operation interface of SBA to reduce the requirements of front-end interface development ability.

Overall structure

For several key points, use the built-in Arthas Spring Boot plug-in of the JVM and refer to the mode of ICBC to establish a perfect client download and modify the script to realize remote control. The amount of development of the built-in scheme is small. Only the relevant open source components need to be integrated to realize the relevant remote use mode and take into account the security. ICBC's scheme is large and complete, which is suitable for the city of exclusive R & D team after the overall architecture planning. The built-in scheme also includes start and stop operations through JMX (the Spring Boot plug-in based on 3.4.5 cannot obtain the relevant handle and cannot be implemented temporarily). It will not be started by default. After the remote JMX is opened, the JVM will add 8 related threads and add about 30MB of virtual machine memory, which is the same as Sba1 The 0 scheme is the same. It is necessary to consider whether the JVM memory can be supported before online startup.

Realization effect

The greatest convenience of SBA 2.0 is to provide the ability to configure and link external web pages. At the same time, if the web page is implemented in the current JVM process, the local permission management of spring security can be realized. In the production environment, the relevant integrated arthas function can be used only after logging in to SBA.

• Login interface

• Embedded connection position

• Use of JMX

• Jump to arthas web console

Transformation scheme

Reference to the original text- Practice of integrating Arthas with SpringBoot Admin Several steps of implementation in.

1. Overall engineering structure

The overall project is modified from the example project of SBA open source project. The specific project link using custom UI is: [spring-boot-admin-sample-custom-ui] _，_ The part in the red box is all the static files of arthas web console, which are entered into the specified directory through the specified configuration of maven resource to realize the custom loading during SBA startup. maven resource configuration – the following:

<resource>
                <directory>static</directory>
                <targetPath>${project.build.directory}/classes/META-INF/spring-boot-admin-server-ui/extensions/arthas
                </targetPath>
                <filtering>false</filtering>
            </resource>

After the URL of the final meta Tom implementation is loaded into the URL of the static meta Tom implementation, it can be included in the URL of the final meta Tom implementation.

2. External link configuration

SBA 2.0 has been in use since then, so it is convenient to expand and integrate. Among them, the official document gives the configuration mode of embedded connection: [Linking / Embedding External Pages].

Refer to the application of sba example project YML configuration:

# tag::customization-external-views[]
    spring:
      boot:
        admin:
          ui:
            external-views:
              - label: "Arthas Console"
                url: http://21.129.49.153:8080/
                order: 1900
    # end::customization-external-views[]

3. Corresponding to the implementation of Spring MVC controller

Refer to the SBA integration part of the original implementation, which is mainly modified to realize the following functions:

• Refresh the instance list loaded by the tunnel server and display it in the AgentID box in the previous paragraph for selection. Click the link.
• Refresh the user-defined IP address (solve the inconsistency between the dual production IP of the production environment and the IP of the operation and maintenance section).

4. Modification and configuration of Arthas spring boot plug-in

Refer to the original SBA integration plug-in modification and client configuration application yml.

The main reason is that the original version of @ onditionbean plugin is automatically loaded through Spring.

The modification is mainly realized by modifying this part. It is not started by default through the configuration file, and then the relevant agent threads are started remotely during use.

5. JMX implementation based on Spring Actuator

SBA client will introduce jolokia core by default in maven Jar. If it is not dependent on SBA client, it can be introduced into the package by itself. It can realize the seamless cooperation between the open http based jmx operation capability through the actor and the related functions of SBA console.

application.yml can open the configuration related to management. According to its own environment, it can also open the Spring security authentication on the client side. SBA can also well support the access of password protected actor endpoint through service discovery.

#Release management
    management:
      endpoints:
        web:
          exposure:
            # Here, * stands for exposing all endpoints only for the purpose of observing the effect. In practice, endpoint exposure is carried out as required
            include: "*"
            exclude: env
      endpoint:
        health:
          # Details are displayed to all users.
          show-details: ALWAYS
      health:
        status:
          http-mapping:
            # The http status code corresponding to the status code returned by the custom health check
            FATAL:  503

JMX implementation refers to the implementation idea of EnvironmentChangeListener in the original text, which can be implemented based on Spring JMX annotation.

@Component
   @ManagedResource(objectName = "com.ArthasAgentManageMbean:name=ArthasMbean", description = "Arthas Remote management Mbean")
   public class ArthasMbeanImpl {
   
       @Autowired
       private Map<String, String> arthasConfigMap;
   
       @Autowired
       private ArthasProperties arthasProperties;
   
       @Autowired
       private ApplicationContext applicationContext;
   
       /**
        * initialization
        *
        * @return
        */
       private ArthasAgent arthasAgentInit() {
           arthasConfigMap = StringUtils.removeDashKey(arthasConfigMap);
           // Prefix all configurations
           Map<String, String> mapWithPrefix = new HashMap<String, String>(arthasConfigMap.size());
           for (Map.Entry<String, String> entry : arthasConfigMap.entrySet()) {
               mapWithPrefix.put("arthas." + entry.getKey(), entry.getValue());
           }
           final ArthasAgent arthasAgent = new ArthasAgent(mapWithPrefix, arthasProperties.getHome(),
                   arthasProperties.isSlientInit(), null);
           arthasAgent.init();
           return arthasAgent;
       }
   
       @ManagedOperation(description = "Get configuration Arthas Tunnel Server address")
       public String getArthasTunnelServerUrl() {
           return arthasProperties.getTunnelServer();
       }
   
       @ManagedOperation(description = "set up Arthas Tunnel Server Address, re attach Effective after")
       @ManagedOperationParameter(name = "tunnelServer", description = "example:ws://127.0.0.1:7777/ws")
       public Boolean setArthasTunnelServerUrl(String tunnelServer) {
           if (tunnelServer == null || tunnelServer.trim().equals("") || tunnelServer.indexOf("ws://") < 0) {
               return false;
           }
           arthasProperties.setTunnelServer(tunnelServer);
           return true;
       }
   
       @ManagedOperation(description = "obtain AgentID")
       public String getAgentId() {
           return arthasProperties.getAgentId();
       }
   
       @ManagedOperation(description = "Get app name")
       public String getAppName() {
           return arthasProperties.getAppName();
       }
   
       @ManagedOperation(description = "obtain ArthasConfigMap")
       public HashMap<String, String> getArthasConfigMap() {
           return (HashMap) arthasConfigMap;
       }
   
       @ManagedOperation(description = "Returns whether it has been loaded Arthas agent")
       public Boolean isArthasAttched() {
           DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory();
           String bean = "arthasAgent";
           if (defaultListableBeanFactory.containsBean(bean)) {
               return true;
           }
           return false;
       }
   
       @ManagedOperation(description = "start-up Arthas agent")
       public Boolean startArthasAgent() {
           DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory();
           String bean = "arthasAgent";
           if (defaultListableBeanFactory.containsBean(bean)) {
               ((ArthasAgent) defaultListableBeanFactory.getBean(bean)).init();
               return true;
           }
           defaultListableBeanFactory.registerSingleton(bean, arthasAgentInit());
           return true;
       }
   
       @ManagedOperation(description = "close Arthas agent，Not implemented yet")
       public Boolean stopArthasAgent() {
           // TODO cannot get the classLoader loaded by the custom tmp folder, so it cannot get com taobao. arthas. core. server. Arthasbootstrap class and call destroy method
           DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory();
           String bean = "arthasAgent";
           if (defaultListableBeanFactory.containsBean(bean)) {
               defaultListableBeanFactory.destroySingleton(bean);
               return true;
           } else {
               return false;
           }
       }
   }

Actual use

After the management project was put into operation, it was used for problem troubleshooting and code hot repair in the production environment for many times. Performance problems are mainly used for online verification and debug ging of performance flow control components and gray release related configuration parameters.

In the early stage of code hot loading, it was operated by jad+mc. Later, it was found that the decompiled code of jad was inconsistent due to environmental configuration and jvm problems. Later, it was solved by packaging and deploying the application source compressed package by maven. It is more reliable to directly use the source built with the same version of application jar for modification. The overall scheme provides effective performance analysis and thermal repair capability in a strictly managed production environment.

remaining problems

Available official com taobao. arthas. agent. attach. The arthasClassLoader and bootstrap class used by the client that starts arthasagent in arthasagent are temporary variables in the method, and the relevant handle cannot be obtained externally to realize the function of closing arthasagent through bootstrap class; The temporary solution is to use the stop command to close the arthas agent in the target process after starting through JMX and using the web console connection.

The existing bytecode loading tool can well implement the online hot deployment and replacement of internal classes and private classes. At the same time, it has been tested to be compatible with skywalk8 X version of the javaagent plug-in, but in the test environment, because the Jacobo coverage collection plug-in is configured, it is incompatible with the Arthas bytecode. When using in some environments, you need to close the corresponding agent before you can normally use the related functions of Arthas.

Welcome to start aliyun. Com mobile hands lab experiences 57 hands-on experiments of Arthas:

Arthas experiment Preview

The 9th Arthas prize essay solicitation will be held from April 1 to April 30, Click to view the essay details.