Distributed log GrayLog usage

Posted by rockinaway on Mon, 21 Feb 2022 12:30:40 +0100

GrayLog brief introduction

GrayLog is a lightweight distributed log management platform, an open source log aggregation, analysis, audit, display and early warning tool. In terms of function, it is similar to ELK, but it is much simpler and lighter than ELK. Relying on the advantages of more concise, efficient and simple deployment and use, it is quickly favored by many companies.

Graylog includes elasticsearch, MongoDb and Graylog modules. Elasticsearch is used to persistently store and retrieve log file data, MongoDb is used to store relevant configurations about Graylog, and Graylog is used to provide Web interface and external interface.

  • Stand alone deployment

  • Cluster deployment

GrayLog function analysis

Simply put, Input represents the source of log data. For logs from different sources, you can use Extractors to convert the log fields. Then, different tag types are used to form different streams, and these log data are stored in the specified Elastic Index library for persistent storage.

Component nameFunction introductionmain features
DashboardsFixed display of data panelIt is mainly used to save the data panel of specific search conditions
SearchingLog information condition searchKeyword search, time search, search save, create panel, group query, result export, query highlight, custom time
AlertSet alarm prompt modeSupport email alarm, HTTP callback and user-defined script triggering
InputsLog data capture and receptionDeploy Sidercar to actively capture or passively report using other services
ExtractorsLog data format conversionjson parsing, kv parsing, timestamp parsing, regular parsing
StreamsClassification and grouping of log informationSet log classification conditions and send them to different index files
IndicesPersistent data storageSet data storage performance
OutputsForwarding of log dataThe parsed Stream is sent to other Graylog clusters or services
PipelinesFiltering of log dataEstablish filtering rules for data cleaning, field addition and deletion, conditional filtering, user-defined functions, etc
SidecarLightweight log collectorEquivalent to C/S mode; Use in large scale
Lookup TablesService resolutionIP based Whois query and source IP based intelligence monitoring
GeolocationVisual geographic locationRealize intelligence monitoring based on source IP

reference resources: https://docs.graylog.org/docs/



Single machine deployment GrayLog

Using docker compose is easy and fast to install. Refer to the official website document Docker - Installing Graylog

Statement: this installation is based on CentOS 8

Docker compose file

version: '3'
      image: mongo:4.2
        - graylog
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
        - http.host=
        - transport.host=localhost
        - network.host=
        - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms512m -Xmx512m"
          soft: -1
          hard: -1
            memory: 1g
        - graylog
      image: graylog/graylog:4.2
        - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
        - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 #Default password: admin
        - GRAYLOG_HTTP_EXTERNAL_URI=http://ip:9000/ 	#  Modify the IP address to your own
        - GRAYLOG_ROOT_TIMEZONE=Asia/Shanghai         #Set time zone
        - GRAYLOG_ALLOW_HIGHLIGHTING=true             #Search highlight
      entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
        - graylog
      restart: always
        - mongo
        - elasticsearch
        - 9000:9000
        - 1514:1514
        - 1514:1514/udp
        - 12201:12201
        - 12201:12201/udp
      driver: bridge

The file is copied from the official website document. I modified three places, namely, the ip and port of accessing the graylog web page, setting the time zone and turning on the highlighting of search content. For more configurations, including custom configuration files and persistent data, please refer to the official documents.

Run the following command in the directory where the docker compose file is located to start graylog

docker-compose up -d  #-d start with daemon

After successful startup, access the ip:port set above to enter the background address of graylog. The default account password is admin/admin

Configure the input, that is, the data source of graylog, and use udp to receive logs. Then click the launch new input button, just enter the Title to save, and other defaults are OK.

spring integrates graylog

Generally, our programs will use logback or other logging frameworks to collect logs. To display our business logs, you only need to write the log data to graylog.

1. Introduce dependency


GELF extension: a log format, which can avoid some problems of syslogs in the traditional sense. The Maven dependency we introduced is to format the log into GELF format and then append it to GrayLog


2. On logback Add the following code to the XML file

<appender name="GELF" class="de.siegmar.logbackgelf.GelfUdpAppender">
    <!-- Graylog Address of the service -->
    <!-- UDP Input port -->
    <!-- maximum GELF Data block size (in bytes), 508 is the recommended minimum and 65467 is the maximum -->
    <!-- Use compression -->
    <encoder class="de.siegmar.logbackgelf.GelfEncoder">
      <!-- Send native log information -->
      <!-- Whether to send the name of the log level. Otherwise, the log level is represented by a number by default -->
      <shortPatternLayout class="ch.qos.logback.classic.PatternLayout">
      <fullPatternLayout class="ch.qos.logback.classic.PatternLayout">
        <pattern>%d - [%thread] %-5level %logger{35} - %msg%n</pattern>

      <!-- Configure the application name (service name) through staticField The tag can customize some fixed log fields -->

Note: change the IP address to your own.

3. Start the program and open the background page of grayog to see the log information.

GrayLog common query syntax

Here are some query syntax commonly used by GrayLog. For more syntax, you can check the official website documents: https://docs.graylog.org/docs/query-language

1. Accurate query by field: full_message:"sys"

2. Query by log level: level_name:"ERROR"

3. Combined multi field query: app_name:"boss" AND full_message:"sys" AND level_name:"DEBUG"

More operations

The functions of GrayLog are far more than these, and more relevant usages can be explored by yourself.


reference resources Official documents

Topics: ElasticSearch Distribution