In this way, the new requirements can be iterated and put on the line safely

Posted by netdynamix on Thu, 24 Feb 2022 08:25:56 +0100

Author: Pan Shengwei (Shimian)

outline

Full link grayscale is one of the core functions of microservices, and it has always been a function that cloud customers must have in the process of microservicing. Full link grayscale involves many technologies and scenarios. If cloud enterprises implement them one by one, they need to spend a lot of manpower to expand and operate them.

Technical fields involved

RPC: routing between microservices
- Java system involves Spring Cloud, Apache Dubbo, mainstream micro service framework, and multilingual Service Mesh
- End cloud interconnection scenario, online traffic DEBUG. The local registers the local service in the registration center through the springboard machine. It is hoped that the online traffic will be routed to the local service after meeting the routing rules, and then DEBUG the instance. The traffic that does not meet the routing rules will be routed to the online instance
MQ
- In the full link pressure test scenario, the pressure test traffic sends a message to the shadow Topic, and the pressure test traffic only subscribes to the shadow Topic
- In the traffic isolation / full link grayscale scenario, using the same Topic, online traffic subscribes to online messages, and isolated traffic subscribes to grayscale messages only
Database
- In the scenario of full link pressure measurement, the pressure measurement flow data falls into the shadow table
- In the scenario of high availability streaming, database operation is prohibited; Under unitization, the traffic has no cell mark, and database operation is prohibited
Redis
- In the scenario of full link pressure measurement, the pressure measurement traffic cache is stored on the shadow KEY
- Cache operation is prohibited in the scenario of high availability streaming; Under unitization, the traffic has no unit flag, and caching is prohibited
Distributed task scheduling
- For task scheduling, the tasks submitted by the gray environment are scheduled to be executed on the machine in the gray environment
Front end
- The page information seen by different customers is inconsistent
Observability
- Monitor the flow direction through observability and check the flow escape

MSE full link grayscale solution

At present, MSE Service Management Professional Edition provides a complete product full link gray-scale solution, covering most scenarios such as RPC, MQ and observability. As long as your architecture is based on Spring Cloud or Dubbo framework, your application can play the enterprise level full link grayscale function without upgrading and one line of code change.

Full link isolated traffic lane

1) the required traffic is' dyed 'by setting traffic rules, and the' dyed 'traffic will be routed to the gray-scale machine.

2) the grayscale traffic carries the grayscale to the downstream to form a grayscale exclusive environment traffic lane. The application without grayscale environment will choose the unmarked baseline environment by default.

End to end stable baseline environment

Unmarked applications belong to the application of baseline stable version, that is, stable online environment. When we will release the corresponding gray version code, then we can configure rules to introduce specific online traffic to control the risk of gray code.

Flow one key dynamic cut-off

After the traffic rules are customized, you can stop and start with one click, add, delete, modify and check according to the demand, and take effect in real time. Gray drainage is more convenient.

Observability

With Lane level single application observability

At the same time, it has the observability of full link applications, and can observe whether there is traffic escape from a global perspective. It's clear at a glance whether it's gray or not.

Low cost access, based on Java Agent technology, without modifying a line of business code

MSE microservice governance capability is based on Java Agent bytecode enhanced technology. It seamlessly supports all Spring Cloud and Dubbo versions on the market for nearly five years. Users can use it without changing a line of code, without changing the existing architecture of the business. It can be up and down at any time without binding. Just open MSE microservice governance professional edition, configure online and take effect in real time.

It has the ability of lossless online and offline, making the release more smooth

After MSE microservice governance is enabled, the application has the ability of lossless online and offline. The release, rollback, capacity expansion, capacity reduction and other scenarios under large traffic can ensure that the traffic is lossless.

How to use MSE full link grayscale

Next, we will demonstrate the ability of full link grayscale. The architecture we use is composed of ingress nginx and the back-end micro service architecture (Spring Cloud). The back-end call link has three hops, shopping cart (a), trading center (b), inventory center (c). The client accesses the back-end service through the client or H5 page, They do service discovery through the Nacos registry.

preparation

Open MSE micro Service Management Professional Edition

Log in to the console of MSE governance center. If you have not opened MSE microservice governance, please open the professional version according to the prompt. If you have opened the basic version of MSE microservice governance, please upgrade to the professional version according to the prompt on the right side of the overview page.

Install ingress nginx component

Log in to the container service console [1].
In the left navigation bar, select market > application catalog.
Enter "ack ingress nginx" in the "application directory" page search box, and clickIcon, and then click components.
On the # details # page, select the # namespace # of the component as # Kube system, and then click # create. After the installation is completed, the "ack ingress nginx default controller" application appears in the namespace "Kube system", indicating that the installation is successful.

Deploy Demo application

Save the following file to ingress gray demo deployment set Yaml, and execute kubectl apply - f ingress gray demo deployment set Yaml} to deploy applications. Here, we will deploy three applications a, B and C. each application will deploy a baseline version and a grayscale version respectively.

# A application base version
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-a
        ports:
        - containerPort: 20001
        livenessProbe:
          tcpSocket:
            port: 20001
          initialDelaySeconds: 10
          periodSeconds: 30

# A apply gray version
---            
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-a-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-a-new
  strategy:
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-a
      labels:
        app: spring-cloud-a-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-a:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-a-new
        ports:
        - containerPort: 20001
        livenessProbe:
          tcpSocket:
            port: 20001
          initialDelaySeconds: 10
          periodSeconds: 30

# B application base version
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-b
  strategy:
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-b
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20002
          initialDelaySeconds: 10
          periodSeconds: 30

# B apply gray version  
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-b-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-b-new
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-b
      labels:
        app: spring-cloud-b-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-b:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-b-new
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20002
          initialDelaySeconds: 10
          periodSeconds: 30

# C application base version
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-c
  template:
    metadata:
      annotations:
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
        imagePullPolicy: Always
        name: spring-cloud-c
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20003
          initialDelaySeconds: 10
          periodSeconds: 30

# C application gray version
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-c-new
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-cloud-c-new
  template:
    metadata:
      annotations:
        alicloud.service.tag: gray
        msePilotCreateAppName: spring-cloud-c
      labels:
        app: spring-cloud-c-new
    spec:
      containers:
      - env:
        - name: JAVA_HOME
          value: /usr/lib/jvm/java-1.8-openjdk/jre
        image: registry.cn-shanghai.aliyuncs.com/yizhan/spring-cloud-c:0.1-SNAPSHOT
        imagePullPolicy: IfNotPresent
        name: spring-cloud-c-new
        ports:
        - containerPort: 8080
        livenessProbe:
          tcpSocket:
            port: 20003
          initialDelaySeconds: 10
          periodSeconds: 30

# Nacos Server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nacos-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nacos-server
  template:
    metadata:
      labels:
        app: nacos-server
    spec:
      containers:
      - env:
        - name: MODE
          value: standalone
        image: nacos/nacos-server:latest
        imagePullPolicy: Always
        name: nacos-server
      dnsPolicy: ClusterFirst
      restartPolicy: Always

# Nacos Server Service configuration
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-server
spec:
  ports:
  - port: 8848
    protocol: TCP
    targetPort: 8848
  selector:
    app: nacos-server
  type: ClusterIP

For portal application a, configure two k8s services. Spring-cloud-a-base corresponds to the base version of a and spring-cloud-a-gray corresponds to the gray version of A.

apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-base
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a

---
apiVersion: v1
kind: Service
metadata:
  name: spring-cloud-a-gray
spec:
  ports:
    - name: http
      port: 20001
      protocol: TCP
      targetPort: 20001
  selector:
    app: spring-cloud-a-new

Fast construction of full link gray scale capability

The swimlane is a set of isolated environments defined for the same version of the application. Only the request traffic that meets the flow control routing rules will be routed to the marking application in the corresponding lane. An application can belong to multiple swimlanes. A swimlane can contain multiple applications. The relationship between applications and swimlanes is many to many.
Lane Group: a collection of lanes. The function of lane group is mainly to distinguish different teams or different scenes.

We can see that we need to realize the capabilities described above. We only need two steps, creating a lane group and creating a lane

Create a swimlane group

Click the create swimlane group button and select the back-end micro service application involved in our swimlane group. According to the above demo, there are three applications: A, B and C

Create Lane

Select the microservice space that is the same as the one when creating the swimlane group at the top of the # full swimlane # page, and then click # at the bottom to create the first shunt swimlane. It should be noted that applications with full link flow control will no longer support Canary publishing, label routing and other functions.

According to the Step of the product, we need to set a lane name, configure the application tag, select the tag associated with the lane, and go to the ACK console to configure the routing rules of Ingress.

Create a completed Lane

There are two modes to view the lane

View mode

Editable mode

Entry Ingress rule

To configure the Ingress rules of the entry, visit www.base.com COM ， route to the base version of a application, visit www.gray.com COM # route to the gray version of a application.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-base
spec:
  rules:
  - host: www.base.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-base
          servicePort: 20001
        path: /

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: spring-cloud-a-gray
spec:
  rules:
  - host: www.gray.com
    http:
      paths:
      - backend:
          serviceName: spring-cloud-a-gray
          servicePort: 20001
        path: /

Verify that the characteristic traffic is routed to the target application

Result verification

Visit www.base.com COM # route to baseline environment

curl -H"Host:www.base.com" http://{ingress-ip}/a
A[172.18.144.155] -> B[172.18.144.120] -> C[172.18.144.79]%

At this point, visit www.gray.com COM # route to gray environment

curl -H"Host:www.gray.com" http://{ingress-ip}/a
Agray[172.18.144.160] -> Bgray[172.18.144.57] -> Cgray[172.18.144.157]%

View the flow monitoring diagram of the marking application.

Select the target lane group on the # full link # page. Select the corresponding application among the applications involved, and the corresponding traffic view will appear

View all application monitoring charts.

In addition to viewing the monitoring chart of a single application, we can also view the monitoring chart of all applications in the swimlane group. By comparing and analyzing the monitoring charts of all applications, more useful information can be analyzed.

You can see which applications are called at the same time.
Analyze the flow escape problem and judge the escape object.

summary

The full link grayscale productization capability of MSE service governance is still evolving. At present, we support MQ, RPC, observability, etc., and more scenarios such as XXL-JOB will be supported later. At present, how do we realize multilingual microservice governance on MSE [3], and use Cloud Toolkit to realize end cloud interconnection of microservices [4] Full link grayscale based on ingress nginx gateway [5], full link grayscale based on MSE cloud native gateway [6], full link grayscale based on self built Spring Cloud Gateway or Zuul gateway [7], full link grayscale based on message queue RocketMQ version [8] With the continuous increase of [Jenkins] and [Jenkins] user iterative development scenarios, we will enrich the best practices related to [Jenkins] and [Jenkins] solutions.

classic case

Call technology

MSE service governance helps our system quickly realize the gray-scale capability of the whole link in a very low-cost and non-invasive way, further improves the stability of our system, and makes the iterative launch of our new requirements more reassuring.

-- call technology architect Tang Changzheng

Since 2014, call technology has entered the field of shared charging, defined and created the industry, and belongs to the earliest shared charging enterprise in the industry. The main business covers self-service leasing of charging treasure, development of customized mall navigation machine, advertising display equipment and advertising communication services. Caller technology has a three-dimensional product line in the industry, large, medium and small cabinets and desktop type. At present, more than 90% of cities in China have implemented business services, with more than 200 million registered users, realizing the needs of users in the whole scene.

Full link gray landing

The business architecture of the incoming call is as follows. The top layer is the user interface such as mobile terminal. The self built Nginx gateway is used as the access layer, and the service layer is a variety of services. Spring Cloud and Dubbo are used as the service framework.

The architecture of calling technology's full link gray landing is as follows:

Configure the traffic diversion configuration on the Nginx layer. 10% of the traffic enters the gray environment, and 90% of the traffic enters the online formal environment without marking. Then the traffic passing through the gray environment will be automatically colored by MSE, so as to carry out the gray routing of the whole link and ensure that the traffic is closed-loop in the gray environment. If there is no machine in the gray environment, For example, if there are only online machines in the payment center, the traffic will go through the online environment. When there are machines with gray-scale environment in our data center, the gray-scale traffic will return to the gray-scale environment of the data center.

tail

MSE's full link gray-scale capability continues to expand and iterate with the deepening of customer scenarios. Only products polished by customers will become more and more lasting. Welcome to have a taste.

Programmer Think