1. Error log alarm practice

1.1. demand

In order to understand the system error in real time more conveniently, I began to look for the alarm solution

1.2. thinking

1.2.1. No bad plan

If there is no shortage of money and a more systematic and perfect solution, the first thing I think of is CAT. It can not only realize error alarm, but also be more intelligent. The error interval of alarm, error alarm content, QPS alarm and other methods are more diversified. It can also view the QPS traffic of interface and so on. However, the fund is limited. Give up

1.2.2. Consider your own implementation

1. The implementation considers whether the log.error method can be intercepted, so various ways to find out whether the logback provides interceptor filters, etc. after checking the official website, it is found that the logback itself provides the way of appender to email, which is very good and directly integrated

1.3. Configuration file

pom

 <dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>janino</artifactId>
    <version>2.7.8</version>
</dependency>
<dependency>
    <groupId>javax.mail</groupId>
    <artifactId>mail</artifactId>
    <version>1.4.7</version>
</dependency>

<configuration>
    <contextName>logback</contextName>
    <!--Parameters in configuration file-->
    <springProperty scope="context" name="applicationName" source="spring.application.name"/>
    <springProperty scope="context" name="alertEmail" source="onegene.alert.email"/>
    <springProperty scope="context" name="profile" source="spring.profiles.active"/>
    <!--  mail -->
    <!-- SMTP server Must be specified. Such as NetEase SMTP The server address is: smtp.163.com -->
    <property name="smtpHost" value="hwhzsmtp.qiye.163.com"/><!--Fill in the smtp server address(ask DBA Or a manager or something)-->
    <!-- SMTP server The port address of. Default: 25 -->
    <property name="smtpPort" value="465"/>
    <!-- Email account, default to null -->
    <property name="username" value="xxxx@163.com.cn"/><!--Sender account-->
    <!-- Send mail password, default is null -->
    <property name="password" value="rVgkwPL4WsWmGV72"/><!--Sender password-->
    <!-- If set to true，appender Will use SSL Connect to the log server. Default value: false -->
    <property name="SSL" value="true"/>
    <!-- Specify which mailbox to send to, multiple<to>Property, specifying multiple destination mailboxes -->
    <property name="email_to" value="${alertEmail}"/><!--Multiple recipient accounts can be separated by commas-->
    <!-- Specifies the sender name. If set to“&lt;ADMIN&gt; "，The sender of the message will be“<ADMIN> " -->
    <property name="email_from" value="xxxx@163.com"/>
    <!-- Appoint emial It needs to meet PatternLayout Format requirements in. If set to“ Log: %logger - %msg "，In the case of email, the title is“[ Error]: com.foo.Bar - Hello World ".  Default value:"%logger{20} - %m". -->
    <property name="email_subject" value="[${applicationName}:${profile}:Error]: %logger"/>
    <!-- ERROR Mail delivery -->
    <appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
        <smtpHost>${smtpHost}</smtpHost>
        <smtpPort>${smtpPort}</smtpPort>
        <username>${username}</username>
        <password>${password}</password>
        <asynchronousSending>true</asynchronousSending>
        <SSL>${SSL}</SSL>
        <to>${email_to}</to>
        <from>${email_from}</from>
        <subject>${email_subject}</subject>
        　　　　 <!-- html format-->
        <layout class="ch.qos.logback.classic.html.HTMLLayout">
            <Pattern>%date%level%thread%logger{0}%line%message</Pattern>
        </layout>
        　　　　 <!-- Here, the level filter is used to specify the level matching before sending -->
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>ACCEPT</onMatch>
            <onMismatch>DENY</onMismatch>
        </filter>
        <cyclicBufferTracker class="ch.qos.logback.core.spi.CyclicBufferTracker">
            <!-- Send only one log entry per email -->
            <bufferSize>1</bufferSize>
        </cyclicBufferTracker>
    </appender>

    <!-- name The value of is the name of the variable, value Is the value defined by the variable. Values defined by are inserted into the logger In context. After defining variables, you can“ ${}"To use variables. -->
    <property name="log.path" value="log"/>

    <!-- Color log -->
    <!-- Color log dependent rendering class -->
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/>
    <conversionRule conversionWord="wex"
                    converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/>
    <conversionRule conversionWord="wEx"
                    converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/>
    <!-- Color log format -->
    <property name="CONSOLE_LOG_PATTERN"
              value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>


    <!--Output to console-->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <!--This log appender It is used for development. Only the lowest level is configured. The log level output by the console is the log information greater than or equal to this level-->
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>debug</level>
        </filter>
        <encoder>
            <Pattern>${CONSOLE_LOG_PATTERN}</Pattern>
            <!-- Set character set -->
            <charset>UTF-8</charset>
        </encoder>
    </appender>

    <!--output to a file-->
    <!-- Time scrolling output level by DEBUG Journal -->
    <appender name="DEBUG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!-- Path and filename of the log file being logged -->
        <file>${log.path}/${applicationName}-log.log</file>
        <!--Log file output format-->
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
            <charset>UTF-8</charset> <!-- Set character set -->
        </encoder>
        <!-- Rolling strategy for loggers, by date, by size -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- Log archiving -->
            <fileNamePattern>${log.path}/${applicationName}-log-%d{yyyyMMdd}.log.%i</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>500MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <!--Log file retention days-->
            <maxHistory>15</maxHistory>
        </rollingPolicy>
    </appender>
    <logger name="com.onegene.platform" level="debug"/>
    <logger name="com.onegene.platform.auth.authority" level="info"/>
    <logger name="org.springframework.security.oauth2.provider.endpoint" additivity="false"/>
    <springProfile name="local">
        <root level="info">
            <appender-ref ref="CONSOLE"/>
            <appender-ref ref="DEBUG_FILE"/>
        </root>
    </springProfile>
    <springProfile name="dev,pro">
        <root level="info">
            <appender-ref ref="CONSOLE"/>
            <appender-ref ref="DEBUG_FILE"/>
            <appender-ref ref="EMAIL"/>
        </root>
    </springProfile>
</configuration>

1.4. Profile interpretation

1. Profile focus

    <springProperty scope="context" name="applicationName" source="spring.application.name"/>
    <springProperty scope="context" name="alertEmail" source="onegene.alert.email"/>
    <springProperty scope="context" name="profile" source="spring.profiles.active"/>

1. I have pulled out most of the extractable variable parameters. This configuration file can be directly put into any project. The log name changes with the spring.application.name parameter in bootstrap.yml
2. The alert sender can also be configured in the configuration file. Note here: the onegene.alert.email and spring.application.name parameters are better configured in bootstrap.yml than application.yml, because the read priority of bootstrap.yml is higher than application.yml, otherwise the two parameters can not be read

At this step, as long as we print the log.error log, we will send all the error logs to the specified email, but this is certainly not enough. We need to cooperate with @ ControllerAdvice to send the log mail as long as an exception is reported. At the same time, we have special needs, such as individual error logs are frequent and inevitable, and do not need to handle, Then we can extend it a little bit, define an interface injection, and deal with whether there is no need to send error mail in the business code

1.5. code

1. exception handling

@ControllerAdvice
@Slf4j
public class SystemExceptionHandler {

    @Autowired(required = false)
    private IExceptionFilter exceptionFilter;

    @ExceptionHandler(value = {DuplicateUniqueException.class, DuplicateKeyException.class})
    @ResponseBody
    public Result duplicateUniqueExceptionExceptionHandler(HttpServletRequest request, Exception e) {
        return getExceptionResult(e, StatusCode.FAILURE_SYSTEM_CODE, "Duplicate unique primary key (or union unique key)", false);
    }

    @ExceptionHandler(value = {FeignException.class, RuntimeException.class})
    @ResponseBody
    public Result FeignExceptionHandler(HttpServletRequest request, Exception e) throws Exception {
        throw e;
    }

    @ExceptionHandler(value = Exception.class)
    @ResponseBody
    public Result commonExceptionHandler(HttpServletRequest request, Exception e) {
        return getExceptionResult(e, StatusCode.FAILURE_CODE, true);
    }


    private Result getExceptionResult(Exception e, int statusCode, boolean ignoreAlert) {
        return getExceptionResult(e, statusCode, e.getMessage(), ignoreAlert);
    }

    private Result getExceptionResult(Exception e, int statusCode, String msg, boolean ignoreAlert) {
        e.printStackTrace();
        String exceptionName = ClassUtils.getShortName(e.getClass());
        StackTraceElement[] stackTrace = e.getStackTrace();
        StringBuilder sb = new StringBuilder();
        for (StackTraceElement stackTraceElement : stackTrace) {
            sb.append(stackTraceElement.toString()).append("\n");
        }
        String message = e.getMessage();
        if (ignoreAlert && filter(e)) {
            log.error("ExceptionName ==> {}，message:{},detail:{}", exceptionName, message, sb.toString());
        }
        return Result.failure(statusCode, msg);
    }

    private boolean filter(Exception e) {
        if (exceptionFilter != null) {
            return exceptionFilter.filter(e);
        }
        return true;
    }
}

The interface is simple

public interface IExceptionFilter {
    boolean filter(Exception e);
}

For exceptions that do not need to be handled here

/**
 * @author: laoliangliang
 * @description: Filter exceptions that do not require an alarm
 * @create: 2020/4/9 10:00
 **/
@Component
@Slf4j
public class FilterAlert implements IExceptionFilter {
    @Override
    public boolean filter(Exception e) {
        if (e instanceof ConnectException) {
            return false;
        }
        return true;
    }
}

1.6. summary

1. So far, the error warning scheme has been fully implemented, and the follow-up is the optimization work. The implementation effect is as follows

Error mailing list

Wrong message content