I. Introduction

Prometheus saves all collected sample data in a time-series manner in a memory database and on a hard disk at regular intervals. Each sample in time series consists of three parts.

• Indicators (metric): metric name and labelsets describing the characteristics of current samples, such as < metric name > {label name >= < label value >,...}; where metric name is named according to the following rules: application name beginning monitoring object numerical type unit
• Time stamp: A time stamp that is accurate to milliseconds.
• Sample value: A float64 floating-point type data represents the current sample value.

Four Data Types of Prometheus

2.1 Counter (counter type)
Counter-type metrics work the same way counters do (unless the system is reset). Counter is generally used for cumulative values, such as recording the number of requests, task completion, and number of errors. Counter has two main methods:

//Add the counter value to 1.
Inc()
// Add the specified value to the counter value and panic if the specified value is less than 0.
Add(float64)

In Prometheus customized metrics monitoring, the use of Counter can be referred to as follows:

public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

    static final Counter requestCounter = Counter.build()
            .name("io_namespace_http_requests_total").labelNames("path", "method", "code") //metric name suggests the end of _total
            .help("Total requests.").register();

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
        String requestURI = request.getRequestURI();
        String method = request.getMethod();
        int status = response.getStatus();

        requestCounter.labels(requestURI, method, String.valueOf(status)).inc(); //Call the inc() function and count + 1 every time a request occurs
        super.afterCompletion(request, response, handler, ex);
    }
}

Counter type data allows users to easily understand the changes in the rate of event generation. The built-in operation functions in PromQL can provide corresponding analysis, such as HTTP application requests.

//Get the growth rate of HTTP requests through the rate() function
rate(http_requests_total[5m])
//Query the top 10 HTTP addresses in the current system
topk(10, http_requests_total)

• 1
• 2
• 3
• 4

2.2 Gauge (dashboard type)
Gauge is an incremental and subtractive indicator class, which can be used to reflect the status of current applications. For example, when monitoring a host, the current content size (node_memory_MemFree) of the host and the available memory size (node_memory_MemAvailable) of the host. Or the container's current cpu usage and memory usage.
Gauge target object mainly includes two methods inc() and dec(), users add or reduce counts.
In Prometheus customized metrics monitoring, the use of Gauge can be referred to as follows:

public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

...Omitted code
static final Gauge inprogressRequests = Gauge.build()
        .name("io_namespace_http_inprogress_requests").labelNames("path", "method", "code")
        .help("Inprogress requests.").register();

@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    ...Omitted code
    inprogressRequests.labels(requestURI, method, String.valueOf(status)).inc();// Counter +1
    return super.preHandle(request, response, handler);
}

@Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
    ...Omitted code
    inprogressRequests.labels(requestURI, method, String.valueOf(status)).dec();// Counter -1
    super.afterCompletion(request, response, handler, ex);
}
}

•  

For Gauge type monitoring indicators, the sample changes over a period of time can be obtained through the PromQL built-in function delta(), such as:

dalta(cpu_temp_celsius{host="zeus"}[2h]) //Calculate the difference of CPU temperature in two hours
predict_linear(node_filesystem_free{job="node"}[1h], 4*3600) //Predict system disk space remaining after 4 hours

2.3 Histogram
Histogram is composed of < basename > bucket {le="< upper inclusive bound>", < basename > bucket {le="+Inf"}, < basename > sum,_count, which is mainly used to sample data over a period of time (usually request duration or response size), and can count its specified interval and total number. Usually, the data collected by Histogram is displayed as a histogram.
In Prometheus customized metrics monitoring, the use of Histgram can be referred to as follows:
Take request response time requests_latency_seconds as an example. For example, we need to record the number of times that http request response time matches in the distribution range {0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10}.

public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

    static final Histogram requestLatencyHistogram = Histogram.build().labelNames("path", "method", "code")
            .name("io_namespace_http_requests_latency_seconds_histogram").help("Request latency in seconds.")
            .register();

    private Histogram.Timer histogramRequestTimer;

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        ...Omitted code
        histogramRequestTimer = requestLatencyHistogram.labels(requestURI, method, String.valueOf(status)).startTimer();
        ...Omitted code
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
        ...Omitted code
        histogramRequestTimer.observeDuration();
        ...Omitted code
    }

• 1
• 2
• 3
• 4

When using Histogram constructor to create Histogram monitoring indicators, the default buckets range is {0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10}. If you want to modify the default buckets, you can use. bukets) Coverage.
Histogram automatically creates three metrics, namely:

• The total number of events, basename_count.

# What it really means: There are currently two http requests
io_namespace_http_requests_latency_seconds_histogram_count{path="/",method="GET",code="200",} 2.0

•  
•  

• The sum of the sizes generated by all events, basename_sum.

# Actual Meaning: The total response time of two http requests is 13.10767080300001 seconds
io_namespace_http_requests_latency_seconds_histogram_sum{path="/",method="GET",code="200",} 13.107670803000001

• 1
• 2

• The number of times the value generated by the event is distributed in the bucket, basename_bucket{le= "upper inclusion"}

•  

2.4 Summary (Summary Type)
Summary type is similar to Histogram type, consisting of < basename > {quantile="<phi>"}, < basename > sum, < basename > count. It is mainly used to represent the results of data sampling over a period of time (usually request duration or response size). It stores quantile data directly, rather than calculated from statistical intervals. Summary differs from Histogram in the following ways:

• Both include < basename > sum and < basename > count;
• Histogram needs to calculate quantile through < basename >_bucket, while Summary stores the value of quantile directly.
  In Prometheus customized metrics monitoring, the use of Summary can be referred to as follows:

public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

    static final Summary requestLatency = Summary.build()
            .name("io_namespace_http_requests_latency_seconds_summary")
            .quantile(0.5, 0.05)
            .quantile(0.9, 0.01)
            .labelNames("path", "method", "code")
            .help("Request latency in seconds.").register();


    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        ...Omitted code
        requestTimer = requestLatency.labels(requestURI, method, String.valueOf(status)).startTimer();
        ...Omitted code
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
        ...Omitted code
        requestTimer.observeDuration();
        ...Omitted code
    }
}

Summary type indicators contain the following data:

• Total number of incidents

# Meaning: The total number of http requests currently occurring is 12
io_namespace_http_requests_latency_seconds_summary_count{path="/",method="GET",code="200",} 12.0

• The sum of the values generated by the event

# Meaning: The total response time of these 12 http requests is 51.029495508s
io_namespace_http_requests_latency_seconds_summary_sum{path="/",method="GET",code="200",} 51.029495508

• 1
• 2

• Distribution of values generated by events

# Meaning: The median response time of these 12 http requests is 3.052404983s
io_namespace_http_requests_latency_seconds_summary{path="/",method="GET",code="200",quantile="0.5",} 3.052404983
# Meaning: The 9-quartile response time of these 12 http requests is 8.003261666s
io_namespace_http_requests_latency_seconds_summary{path="/",method="GET",code="200",quantile="0.9",} 8.003261666

Reference resources

1.Prometheus Metric Type
2.Custom Metrics: Let Prometheus monitor your application (Spring version)
3.Custom Metrics: Let Prometheus monitor your application
4.Using Prometheus+Grafana to monitor MySQL practice
5.Comprehensive Learning Prometheus
6.Prometheus Notes (1) metric type