Preface

The default log driver for docker is json-file Every container will generate a / var/lib/docker/containers/containerID/containerID-json.log locally. The log driver supports extension. This chapter focuses on the fluent driver collecting docker logs

Fluentd is an open source data collector for unified logging layer. It is the sixth CNCF graduation project after Kubernetes, Prometheus, envy, CoreDNS and containerd. It is commonly used to compare with elastic's logstash. Relatively speaking, fluentd is lighter and more flexible. Now the development of fast community is very active. When writing this blog, the star of github is 8.8k, and the fork is 1k

premise

1. docker
2. understand fluentd To configure
3. docker-compose

Prepare profile

docker-compose.yml

version: '3.7'

x-logging:
  &default-logging
  driver: fluentd
  options:
    fluentd-address: localhost:24224
    fluentd-async-connect: 'true'
    mode: non-blocking
    max-buffer-size: 4m
    tag: "kafeidou.{{.Name}}"  #Configure the tag of the container with kafeidou. As the prefix, and the container name as the suffix. Docker compose will add the copy suffix to the container, such as fluent 1

services:

  fluentd:
   image: fluent/fluentd:v1.3.2
   ports:
    - 24224:24224
   volumes:
    - ./:/fluentd/etc
    - /var/log/fluentd:/var/log/fluentd
   environment:
    - FLUENTD_CONF=fluentd.conf

  fluentd-worker:
   image: fluent/fluentd:v1.3.2
   depends_on:
     - fluentd
   logging: *default-logging

fluentd.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match kafeidou.*>
  @type              file
  path               /var/log/fluentd/kafeidou/${tag[1]}
  append             true
  <format>
    @type            single_value
    message_key      log
  </format>
  <buffer tag,time>
    @type             file
    timekey           1d
    timekey_wait      10m
    flush_mode        interval
    flush_interval    5s
  </buffer>
</match>

<match **>
  @type              file
  path               /var/log/fluentd/${tag}
  append             true
  <format>
    @type            single_value
    message_key      log
  </format>
  <buffer tag,time>
    @type             file
    timekey           1d
    timekey_wait      10m
    flush_mode        interval
    flush_interval    5s
  </buffer>
</match>

Because fluent D needs to have write permission in the configured directory, it needs to prepare the directory where log is stored and give permission first
Create directory

mkdir /var/log/fluentd

Give permission, here for experiment demonstration, direct authorization 777

chmod -R 777 /var/log/fluentd

Execute the command in the directory of docker-compose.yml and fluent.conf:
docker-compose up -d

[root@master log]# docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "log_default" with the default driver
Creating fluentd ... done
Creating fluentd-worker ... done

Check the log directory. There should be a corresponding container log file:

[root@master log]# ls /var/log/fluentd/kafeidou/
fluentd-worker.20200215.log  ${tag[1]}

This is my last experiment result. I will create a ${tag[1]} directory, which is quite strange, and there will be two files in this directory

[root@master log]# ls /var/log/fluentd/kafeidou/\$\{tag\[1\]\}/
buffer.b59ea0804f0c1f8b6206cf76aacf52fb0.log  buffer.b59ea0804f0c1f8b6206cf76aacf52fb0.log.meta

If you understand this, you are welcome to communicate!

Architecture summary

Why not use docker's original log?

Let's first look at the structure of the original docker log:

Docker will generate a log file for each container in / var/lib/docker/containers/containerID/containerID-json.log path of the local machine to store the docker logs

In the figure above, there are 7 containers in total. If there are 7 microservices, it is inconvenient to view the logs. In the worst case, you need to view the logs of each container on three machines

What's different with fluent D?

After using fluent to collect docker logs, you can summarize the containers together. Take a look at the architecture after configuring the fluent configuration file of this article:

Because fluent D is configured to be stored in the local directory of the machine where fluent D is located, the effect is to collect the container logs of other machines into the local directory of the machine where fluent D is located

Can fluent only collect container logs locally?

In fact, fluent can transfer the collected logs out again, for example, to the storage software such as elastic search

Fluent flexibility

There are many other things that fluent D can do. Fluent D itself can be a transport node as well as a receiving node. It can also filter specific logs, format specific content logs, and transmit the matching specific logs again. Here is just a simple effect of collecting docker container logs