In earlier versions, Kubernetes provided a combination of heapster, influxDB and grafana to monitor and control systems. Now the more popular monitoring tool is prometheus, which is an open source version of Google's internal monitoring and alarm system.

Compared with other traditional monitoring tools, Prometheus has the following characteristics:
Multidimensional data model with time series data identified by metric name and key/value pair
Have a flexible query language
It does not depend on distributed storage, but only on local disks.
Retrieving Time Series Data through HTTP Services
Push is also supported to add time series data
It also supports target discovery through service discovery or static configuration
Multiple graphics and dashboard support

Prometheus consists of several components, but many of them are optional:
Prometheus Server: Used to capture indicators and store time series data
exporter: Exposing Indicators to Tasks
Push gateway: Push way to push index data to the gateway
Alert manager: alarm component for handling alarms
adhoc: for data query

1. Create a separate namespace

apiVersion: v1
kind: Namespace
metadata:
  name: kube-ops

2. Manage the configuration file prometheus.yml in the form of configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-ops
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      scrape_timeout: 15s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']

The configuration file prometheus.yml contains three modules: global, rule_files, and scrape_configs
The global module controls the global configuration of Prometheus Server
The rule_files module defines the location of the rules. prometheus can load the rules according to this configuration to generate new time series data or alarm information. At present, we do not configure any rules.
scrape_configs is used to control which resources prometheus monitors.
In the default configuration, there is a separate job called prometheus, which collects time series data of the Prometheus service itself. This job contains a single, static configuration goal: listening for port 9090 on localhost.
prometheus collects metrics by default through the target / metrics path. So, the default job is through the URL: http://localhost:9090/metrics acquisition metrics.
3. Configuring rbac authentication

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-ops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: kube-ops

4. Configure pv and pvc for data persistence

apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    server: 192.168.1.244
    path: /data/k8s

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus
  namespace: kube-ops
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

5. Create the Pod resource of prometheus
$ docker pull prom/prometheus:v2.4.3

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
  namespace: kube-ops
  labels:
    app: prometheus
spec:
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - image: prom/prometheus:v2.4.3
        name: prometheus
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention=24h"
        - "--web.enable-admin-api"  # Control access to admin HTTP API, including deletion of time series and other functions
        - "--web.enable-lifecycle"  # Supporting hot updates, directly execute localhost:9090/-/reload with immediate effect
        ports:
        - containerPort: 9090
          protocol: TCP
          name: http
        volumeMounts:
        - mountPath: "/prometheus"
          subPath: prometheus
          name: data
        - mountPath: "/etc/prometheus"
          name: config-volume
        resources:
          requests:
            cpu: 100m
            memory: 512Mi
          limits:
            cpu: 100m
            memory: 512Mi
      securityContext:
        runAsUser: 0
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: prometheus
      - configMap:
          name: prometheus-config
        name: config-volume

$ kubectl get pod -n kube-ops
prometheus-77d968648-w5j6z 1/1 Running 53 82d
6. Create svc of prometheus pod

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: kube-ops
  labels:
    app: prometheus
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
    - name: web
      port: 9090
      targetPort: http

$ kubectl get svc -n kube-ops
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
http://192.168.1.243:32619
Click status - targets to view the status of the monitoring directory