Configuration in Sysctl in Kuberntes (PHP FPM concurrency can only be 300)

Posted by rxero on Sat, 18 Dec 2021 02:26:57 +0100

Background:

Deploy applications in kubernetes cluster and conduct stress testing on applications. jmeter's stress test is about 300 requests per second (18000 requests are collected in elasticsearch every minute). Check the log with nginx's error log:

But my cpu memory resources are not full. Through the search engine, it is found that the environment is basically similar to that of the following blogs, and PHP FPM is also a socket:

See: http://www.bubuko.com/infodetail-3600189.html

solve the problem:

Modify net core. somaxconn

Enter your nginx PHP container to view:

bash-5.0# cat /proc/sys/net/core/somaxconn
128

Randomly find a work node to view the somaxconn of the host:

root@ap-shanghai-k8s-node-1:~# cat /proc/sys/net/core/somaxconn
32768

Note: This is a tke cluster. All parameters are default. Not modified
Next, modify the application configuration file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: paper-miniprogram
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: paper-miniprogram
  template:
    metadata:
      labels:
        app: paper-miniprogram
    spec:
      containers:
        - name: paper-miniprogram
          image: ccr.ccs.tencentyun.com/xxxx/paper-miniprogram:{data}
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "1024M"
              cpu: "1000m"
            limits:
              memory: "1024M"
              cpu: "1000m" 
      imagePullSecrets:                                              
        - name: tencent
---

apiVersion: v1
kind: Service
metadata:
  name: paper-miniprogram
  labels:
    app: paper-miniprogram
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: paper-miniprogram

Amend as follows:
Add initContainers configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: paper-miniprogram
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: paper-miniprogram
  template:
    metadata:
      labels:
        app: paper-miniprogram
    spec:
      containers:
        - name: paper-miniprogram
          image: ccr.ccs.tencentyun.com/xxxx/paper-miniprogram:{data}
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "1024M"
              cpu: "1000m"
            limits:
              memory: "1024M"
              cpu: "1000m" 
      initContainers:
      - image: busybox
        command:
        - sh
        - -c
        - echo 1000 > /proc/sys/net/core/somaxconn
        imagePullPolicy: Always
        name: setsysctl
        securityContext:
          privileged: true
      imagePullSecrets:                                              
        - name: tencent
---

apiVersion: v1
kind: Service
metadata:
  name: paper-miniprogram
  labels:
    app: paper-miniprogram
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: paper-miniprogram

php-fpm listen. Modify the backlog parameter

First look at the system variable net ipv4. tcp_ max_ syn_ Parameter value of backlog

cat /proc/sys/net/core/netdev_max_backlog
#OR
sysctl -a|grep backlog

Then take a look at listen in php. Configuration of backlog:

511 let's 511 first. No modification first. If you modify this value, you also need to modify the privileged mode. Net in the container ipv4. tcp_ max_ syn_ Value of backlog?

Official about sysctl

kubernetes has official usage instructions for syscl: https://kubernetes.io/zh/docs/tasks/administer-cluster/sysctl-cluster/

Then the sequelae of doing so:

Personally, I think the privileged mode will bring security and other problems, but I still don't like pod to enable the privileged mode.

Personally, I think it's a better way:

Through grafana Kanban, it is found that the resource utilization rate of pod is still not so high. Reasonably adjust the resource limits parameter.

Enable hpa horizontal autoscale.
To sum up, I also want to keep the default net core. somaxconn=128. And rely on more copies to meet the high load. This is also in line with the idea of using containers.
The key is that many people think that expanding resources can increase concurrent load is wrong It is better to tune parameters.