Canary release of front-end and back-end applications based on Kubernetes

The company's R & D management platform implements Devops of Gitlab+Kubernetes. In the ToB and ToC scenarios, due to the large number of users and more or less differences between the pre release environment and the production environment, there are still many uncertainties and great risks when the production environment releases the version. Therefore, the demand side puts forward the demand to support Canary release. There are many Canary release schemes. The following are two commonly used schemes.

1. Implementation of Canary release with Deployment rolling update strategy

Using the rolling update strategy maxSurge and maxUnavailable of Deployment to set the maximum number of nodes that can exceed the expectation and the maximum number of unavailable nodes, a simple Canary release can be realized.
rollingUpdate.maxSurge the maximum number of nodes that can exceed the expectation, the percentage is 10% or the absolute value is 5
rollingUpdate.maxUnavailable maximum number of unavailable nodes, percentage or absolute value

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
    name: demo-deployment
    namespace: default
spec:
  replicas: 10
  selector:
    matchLabels:
      name: hello-deployment
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0     
  template:
    metadata:
      labels:
        name: demo-deployment
    spec:
      containers:
      - name: webserver
        image: nginx:1.14
        ports:
        - containerPort:80

This scheme is only suitable for Canary publishing of a single application. If it is a front-end and back-end application, it will be inappropriate. The front-end normal version will call the back-end Canary version, or the front-end Canary version will call the back-end normal version. So Pass dropped the plan and found another way.

2. Progress nginx configuring progress annotations for Canary Publishing

Ingress nginx supports configuring Ingress Annotations to implement Canary publishing in different scenarios. Nginx Annotations supports the following four Canary rules:

• nginx.ingress.kubernetes.io/canary-by-header: traffic segmentation based on Request Header, suitable for gray publishing and A/B testing. When the Request Header is set to always, the request will be sent to the Canary version; When the Request Header is set to never, the request will not be sent to the Canary entry; For any other Header value, the Header is ignored and the request is compared with other Canary rules by priority.
• nginx.ingress.kubernetes.io/canary-by-header-value: the value of the Request Header to be matched, which is used to inform Ingress to route the request to the service specified in Canary Ingress. When the Request Header is set to this value, it will be routed to the Canary entry. This rule allows users to customize the value of the Request Header, which must be used together with the previous annotation (i.e. Canary by header).
• nginx.ingress.kubernetes.io/canary-weight: traffic segmentation based on service weight. It is applicable to blue-green deployment. The weight range is 0-100. Requests are routed to the services specified in Canary Ingress by percentage. A weight of 0 means that the Canary rule will not send any requests to the services of Canary Ingress. A weight of 100 means that all requests will be sent to Canary Ingress Mouth.
• nginx.ingress.kubernetes.io/canary-by-cookie: Cookie based traffic segmentation, applicable to grayscale publishing and A/B testing. It is used to inform Ingress to route requests to the service specified in Canary Ingress. When the cookie value is set to always, it will be routed to the Canary entry; when the cookie value is set to never, requests will not be sent to the Canary entry; For any other value, the cookie is ignored and the request is compared with the priority of other Canary rules.
  
  Note: Canary rules are sorted by priority as follows:
  canary-by-header - > canary-by-cookie - > canary-weight
  Obviously, Canary weight is also a random strategy, which will also cause the front-end normal version to call the back-end Canary version, so Pass drops this strategy.
  Canary by cookie and Canary by header value are suitable for the back-end Canary publishing implementation, and Canary by cookie is suitable for the front-end Canary publishing implementation.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-canary
  annotations:    
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "canary"
    nginx.ingress.kubernetes.io/canary-by-cookie: "canary"
spec:
  rules:
  - host: demo-api.fulu.com
    http:
      paths:
      - backend:
          serviceName: demo-api-canary
          servicePort: 80

The front end sets the value of the cookie canary=always in the browser according to the user tag or manually. If the front end code detects this cookie, it will pass the request header of canary=always to the back end, so that the front end normal version can access the back-end normal version, or the front-end canary version can access the back-end Canary version, without version confusion.

2.1 process

• If it is the first release, it must be a normal version.
• If the canary version is released, first check whether there are ingress, service and deployment with - canary suffix. If so, replace the image of the canary version. If not, create ingress, service and deployment with - canary suffix.
• If the normal version is published, first check whether there are ingress, service and deployment with - canary suffix. If so, delete them first, and then replace the image of the normal version.
  

2.2 code

Front end code transformation

Take React as an example, modify the axios request interceptor to obtain whether to access the Canary version from the cookie. If so, pass the Canary version request header to the back-end service.

import cookie from 'react-cookies'

axios.interceptors.request.use(
  (config) => {
    // Canary Channel 
    const canaryCookie = cookie.load('canary')
    if (canaryCookie !== undefined) {
      config.headers.canary = canaryCookie
}
return config
  },
  (error) => {
    return Promise.reject(error)
  }
)

Backend code transformation

Take. Net as an example, transparently transmit canary request headers to other Api services through proxy

public class CanaryHttpMessageHandler : DelegatingHandler
{
    private readonly IHttpContextAccessor _httpContextAccessor;
    private const string Canary = "canary";
    public CanaryHttpMessageHandler(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, 
CancellationToken cancellationToken)
    {
        var headers = _httpContextAccessor?.HttpContext?.Request?.Headers;
        if (headers != null && headers.ContainsKey(Canary))
        {
            // Transparent request header canary
            var canary = headers[Canary].ToString() ?? "";
            if (!string.IsNullOrWhiteSpace(canary))
                request.Headers.TryAddWithoutValidation(Canary, canary);
        }
        return await base.SendAsync(request, cancellationToken);
    }
}

Front end Chrome test

Use the Chrome browser to access the front-end website F12, and enter document.cookie='canary 'in the Console=
always' manually set the cookie value of canary.

You can see the settings in Cookies. At this time, the gray version is used to access the front-end website.

If the cookie of canary is deleted, it is normal to visit the front-end website at this time

Backend Postman test

Use Postman to access the back-end interface and enter canary = always in the request header. At this time, the access back-end service is grayscale version.

If the request header of canary is deleted, the back-end service access is the normal version

last

Overall, progress annotations meets our requirements. If we want to further implement the user tag to determine whether users access the Canary version, we still need to continue to iterate, which is not difficult. As for the implementation of Canary release, istio also supports it. It belongs to the infrastructure level and is less invasive to the code. In addition, due to The time is short and the writing is not very detailed, but the core idea and code are on it. I hope it can help you!