background
In the monitoring alert project, prometheus and alertmanager are used to implement the monitoring alert. Common indicators such as cpu are monitored by node_exporter, network connectivity is monitored by blackbox_exporter, etc. In the actual test, different network interrupts are constructed and service interruptions are cumbersome. So a custom collector is added based on node_exporter to read from the file.This makes it easy to configure data for alarm routing.
Implementation
Embedding node_exporter custom indicators is easy, just define three functions (registration function, collector function, and update indicator function).
package collector
import (
"github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/prometheus/client_golang/prometheus"
"gopkg.in/yaml.v2"
"io/ioutil"
)
const (
probeSubsystem = "probe"
)
type probeCollector struct {
probeSuccess *prometheus.Desc
logger log.Logger
}
// Customized probe metrics used to simulate icmp and tcp probes similar to those in blackbox_exporter
type ProbeConfig struct {
Icmp []struct {
Targets []string `yaml:"targets"`
Success bool `yaml:"success"`
} `yaml:"icmp,omitempty"`
Tcp []struct {
Targets []string `yaml:"targets"`
Success bool `yaml:"success,omitempty"`
} `yaml:"tcp,omitempty"`
}
func init() {
// Registration function for registering ProbeCollector to exporter for indicator collection
registerCollector("probe", defaultEnabled, NewProbeCollector)
}
// Build a collector function that returns a custom collector that requires an Update interface to complete collection of custom indicators
func NewProbeCollector(logger log.Logger) (Collector, error) {
tmpProbe := prometheus.NewDesc(
prometheus.BuildFQName(namespace, probeSubsystem, "success"),
"Whether probe(icmp, tcp etc.) success or not.",
[]string{"module", "target"}, nil,
)
return &probeCollector{
probeSuccess: tmpProbe,
logger: logger,
}, nil
}
// Update interface for collector to update custom metrics, data collected by prometheus based on time intervals defined by it
func (p *probeCollector) Update(ch chan<- prometheus.Metric) error {
// By reading the configuration file, changing the collected metrics properties by modifying the configuration file, FakeDataFile is the file name, and init function is initialized in collector.go
data, err := ioutil.ReadFile(FakeDataFile)
if err != nil {
level.Error(p.logger).Log("error", err.Error())
return err
}
ret := ProbeConfig{}
err = yaml.Unmarshal(data, &ret)
if err != nil {
level.Error(p.logger).Log("error", err.Error())
return err
}
var ifSuccess float64
if ret.Icmp != nil {
for _, v := range ret.Icmp {
if v.Success {
ifSuccess = 1
} else {
ifSuccess = 0
}
for _, target := range v.Targets {
ch <- prometheus.MustNewConstMetric(p.probeSuccess, prometheus.GaugeValue, ifSuccess, "icmp", target)
}
}
}
if ret.Tcp != nil {
for _, v := range ret.Tcp {
if v.Success {
ifSuccess = 1
} else {
ifSuccess = 0
}
for _, target := range v.Targets {
ch <- prometheus.MustNewConstMetric(p.probeSuccess, prometheus.GaugeValue, ifSuccess, "tcp", target)
}
}
}
return nil
}
The three important functions are the registerCollector function in the init function, the collector function NewProbeCollector, and the collector's update interface Update.
Below are configuration items related to probe probe metrics
# This config file should be under local path or "/opt/" path
# ......
# TCP/IP
icmp:
- targets:
- 127.0.0.1
- 10.0.0.0
success: true
- targets:
- 10.0.0.1
success: false
tcp:
- targets:
- 127.0.0.1:8080
success: true
- targets:
- 127.0.0.1:80
success: false
# ......
Visit the relevant webpage to see the following metrics information
# HELP node_probe_success Whether probe(icmp, tcp etc.) success or not.
# TYPE node_probe_success gauge
node_probe_success{module="icmp",target="10.0.0.0"} 1
node_probe_success{module="icmp",target="10.0.0.1"} 0
node_probe_success{module="icmp",target="127.0.0.1"} 1
node_probe_success{module="tcp",target="127.0.0.1:80"} 0
node_probe_success{module="tcp",target="127.0.0.1:8080"} 1
Problems encountered
Invocation of init function
The parameter in the node_exporter.go file specifies the profile path, but the global variable in the collector/collector.go does not receive the value of the parameter. This is because the init function of the package executes before the main function executes. Set the global variable in the package collector by calling the Initialization function manually instead.
Related code in node_exporter.go:
func main() {
var (
// ......
fakeDataConfigFile = kingpin.Flag(
"fakedata.config-file",
"Config file which includes fake data to collect.",
).Default("/opt/config_fake_data.yaml").String()
)
// ......
if *disableDefaultCollectors {
collector.DisableDefaultCollectors()
}
// Assign the value received by the node_exporter main function parameter to the global variable FakeDataFile in the package collector and manually call the Initialization function InitCollector() of the collector package
collector.FakeDataFile = *fakeDataConfigFile
collector.InitCollector()
// ......
}
Initialization function in collector.go
func InitCollector() {
loadConfig()
setDefaultConfig()
updateConfig()
watchConfig()
}
Loading configuration (loadConfig), setting default values for configuration items (setDefaultConfig), updating configuration items (updateConfig), and monitoring configuration files (watchConfig) are done in the InitCollector of the collector package, mainly using the viper library:
- "github.com/fsnotify/fsnotify"
- "github.com/spf13/viper"
About the execution order of init functions:
- Prioritize execution of init functions in dependent packages
- In program execution, the init function of a package executes only once
- The same package, init functions in different source files are executed in order from small to large source file names
- Multiple init functions in the same source file, executed in the order defined
With this customized monitoring indicator, the indicator is registered in the initialization function, but the global variable FakeDataFile in the package is assigned after all initialization is completed. However, since the configuration file is read only when the indicator's value is updated, it has no effect on the update of the indicator.
Compilation control of go
A label for compilation control is included above the package name in the collector's other source files, such as the cpu_linux.go file
// ...... // +build !nocpu package collector //......
When implementing custom metrics, there is no need to include a label for compilation control unless specifically needed.
In addition, if you only have the // +build field without any labels, the source file will also be ignored when you execute go build.
In addition to compilation control through tags, compilation control can also be achieved through file suffixes, such as cpu_linux.go.
For more compilation control of go, refer to: