In daily monitoring, health monitoring is very important for services. Most of its monitoring methods are to monitor service ports and service processes, usually only one of them needs to be monitored.

blackbox_exporter is one of the official exporters of Prometheus, which can provide monitoring data collection of http, dns, tcp and icmp

Blackbox? Exporter application scenario

HTTP test
Define Request Header information
Determine HTTP status / HTTP responses header / HTTP body content
TCP test
Business component port status monitoring
Application layer protocol definition and monitoring
ICMP test
Host detection mechanism
POST test
Interface connectivity
SSL certificate expiration time
1 install blackbox

cd /usr/local
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar xf blackbox_exporter-0.16.0.linux-amd64.tar.gz 
mv blackbox_exporter-0.16.0.linux-amd64 blackbox_exporter

2 register blackbox ﹐ exporter as a service

$ cat  /etc/systemd/system/blackbox_exporter.service
[Service]
Restart=on-failure
WorkingDirectory=/usr/local/blackbox_exporter/
ExecStart=/usr/local/blackbox_exporter/blackbox_exporter --config.file=/usr/local/blackbox_exporter/blackbox.yml
[Install]
WantedBy=multi-user.target

3 start service

systemctl start     blackbox_exporter
sudo systemctl status  blackbox_exporter
sudo systemctl enable  blackbox_exporter

//The service port is 9115

4 monitoring service port example

cat  /usr/local/prometheus/prometheus.yml 
 - job_name: 'check_nginx_port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['10.0.2.100:80']
        labels:
          instance: 'port_status'
          app: 'tcp_80'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.0.2.100:9115

5 example of monitoring domain name

cat  /usr/local/prometheus/prometheus.yml 
 - job_name: 'check_domain_status'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://test.com']
        labels:
          instance: 'domain_status'
          app: 'web_domain'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.0.2.100:9115

6 ping detection

cat  /usr/local/prometheus/prometheus.yml 
 - job_name: 'check_domain_status'
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['10.0.2.100']
        labels:
          instance: 'node_status'
          app: 'node'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 10.0.2.100:9115

7 main test port detection today

Configure prometheus

cat  /usr/local/prometheus/prometheus.yml 
 - job_name: 'check_nginx_port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    file_sd_configs:
    - files: ['/usr/local/prometheus/conf.d/check80.json']
      refresh_interval: 60s
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.0.2.100:9115

//Dynamic file registration


$ cat /usr/local/prometheus/conf.d/check80.json 
[
{
     "targets":[
            "10.0.2.101:80"
            ],
    "labels": {
       "hostname":"front-01",
       "app": "tcp_80"
       }
},
{
     "targets":[
            "10.0.2.102:80"
            ],
    "labels": {
       "hostname":"front-02",
       "app": "tcp_80"
       }
}


]

Configure alarm rules

$ cat /usr/local/prometheus/rules/nginxservice_rules.yml 
groups:
- name: nginxservices
  rules:
  - alert: 80_port is down
    expr: probe_success{job=~"check_nginx_port_status"} == 0 
    for: 1m
    labels:
     severity: 3 
    annotations:
     summery: "The current value is: {{ $value }}"
     console: 'Host {{ $labels.hostname }}, nginx The server is down!'

Restart prometheus

systemctl restart prometheus

At this time, we configure the detected port and server, and configure the alarm rules.

At the same time, grafana also has the corresponding template 9965