Implementation tutorial of Java project CI/CD

What is Ci? What is the difference between CI and CD?

The acronym CI/CD has several different meanings. "Ci" in CI/CD always refers to continuous integration, which belongs to the automated process of developers. A successful CI means that new changes to the application code are built, tested, and merged into a shared repository on a regular basis. This solution can solve the problem that there are too many application branches in one development, resulting in conflicts.

"CD" in CI/CD refers to continuous delivery and / or continuous deployment, and these related concepts are sometimes used interchangeably. Both are related to the automation of subsequent stages of the pipeline, but they are sometimes used separately to illustrate the degree of automation.

Continuous delivery usually means that developers automatically error test and upload changes to the repository (e.g GitHub Or container Registry), and then the operation and maintenance team will deploy it to the real-time production environment. This aims to solve the problem of poor visibility and communication between development and operation and maintenance teams. Therefore, the purpose of continuous delivery is to ensure that the workload required to deploy new code is minimized.

Continuous deployment (another kind of "CD") refers to automatically publishing developer changes from the repository to the production environment for use by customers. It is mainly to solve the problem that the manual process reduces the application delivery speed, resulting in overload of the operation and maintenance team. Continuous deployment is based on the advantages of continuous delivery and realizes the automation of subsequent stages of the pipeline.

CI/CD may only refer to the related links composed of continuous integration and continuous delivery, or the related links composed of continuous integration, continuous delivery and continuous deployment. More complicated, sometimes "continuous delivery" also includes continuous deployment process.

In the final analysis, we don't need to worry about these semantics. You just need to remember that CI/CD is actually a process (usually vividly expressed as a pipeline) to achieve a high degree of continuous automation and continuous monitoring in application development. Depending on the case, the specific meaning of the term depends on the degree of automation of the CI/CD pipeline. Many enterprises start by adding CIS and then gradually automate delivery and deployment (for example, as Cloud native application Part of).

From RadHat:https://www.redhat.com/zh/topics/devops/what-is-ci-cd , see this document for more information.

In a word, I think it should be: mechanical things let machines do. A development team without CI/CD may be like this: unable to manage code, many people cooperate (git repository is also a part of CI), a series of shell s need to be processed manually, and the release of code needs to log in to the server, etc; On the contrary, with CI/CD, these things are left to the machine to complete, reducing the probability of manual operation errors, and freeing up fragment time can also do more meaningful things.

What is the ideal CI/CD development flow?

I think the ideal CI/CD development flow should include three stages: build, deploy and notify. The build phase focuses on code construction and unit testing, the deploy phase focuses on code deployment in the test/gray/prod environment, and the notify phase focuses on online notification, as shown in the following figure;

The following content completes the deployment from 0 to 1 around the two phases of build and deploy. (excluding merged branches, take the app end of yiyi.com as an example)

1. Gitlab runner installation and registration

I use gitlab runner here. The document: https://docs.gitlab.com/runner/

2. Secret free login

Why do I need password free login?

• The dist file generated by the build should be transferred to the target server (tester / generator), either based on http network protocol or ssh protocol (or other file transfer protocols?)
• The file receiving interface needs to be written based on http. Here, the file receiving interface based on ssh is directly used rsync , simple and safe!
• A series of script s defined in the runner are executed in a docker container and cannot be manually intervened, so the login server must be made secret free.

First, configure a password free login server on the local machine (the machine where the runner is registered):

Generate a pair of public and private keys

Use rsa as asymmetric encryption:

ssh-keygen

Note: the command will be in ~ / Production public key and private key under ssh directory

Define ssh config content

At ~ / The ssh/config file writes the following contents (the file does not exist and is created directly):

Host any_name
  HostName server_ip
  User user
  IdentityFile ~/.ssh/id_rsa

Multiple hosts can be written as follows:

Host lyg-dev
    Port 22
    HostName ip
    User root
    IdentityFile ~/.ssh/id_rsa

Host project-dev
	Port 22
  	HostName ip 
  	User root
  	IdentityFile ~/.ssh/id_rsa

Note: the ssh config file is defined for quick access. Just like you configure host, without hostname, you can only access ip. After configuration, you can use ssh any_name logged in to the server. Of course, not surprisingly, you will be asked to enter the login password of the server.

Password free login

The essence of secret free login is to store the local public key in the authorized of the target server_ Keys file (the file does not exist on the server and can be created directly.)

ssh-copy-id -i ~/.ssh/id_rsa.pub username@ip

In particular: if your port is not the default 22 port, add the port number - p PORT

Verify login

ssh any_name

Not surprisingly, you should be able to log in to the server directly. So, let's go back to the configuration of gitlab

3. Define ssh configuration information on gitlab

We enter the gitlab page and define it in Settings > > CI / CD > > environment variables gitlab. Several variables that appear on YML:

• SSH_PRIVATE_KEY: copy the private key of the local machine (the machine where the runner is located): ~ / ssh/id_rsa
• SSH_CONFIG: copy the information just defined by ssh config: ~ / ssh/config
• SSH_KNOWN_HOST `: this variable is defined to allow ssh to authenticate the server (otherwise it will be considered as an untrusted environment by ssh). The variable value is generated using the following command:

ssh-keyscan -p PORT IP

Then open ~ / ssh/known_hosts to save the content corresponding to the secondary ip to SSH_KNOWN_HOST

SSH_USER SSH_HOST_DEV these two values are optional.

4. Use rsync to transfer application files

rsync -rve ssh target/app.jar root@project-dev:/project/code/project/app/app

Note: Project dev is the Host value defined in ssh config.

5. Start the project

The following commands are used for the test environment. The official yiyi.com environment needs to judge the projects and ports of the old version, which are complex and will not be written for the time being.

Use SSH to connect to the server remotely, enter the corresponding directory, and then execute shutdown SH and startup jar sh. Note that it is startup jar

    ssh -o StrictHostKeyChecking=no -T project-dev "cd /project/code/project/ && sh shutdown.sh app"
    ssh -o StrictHostKeyChecking=no -T project-dev "source /etc/profile && cd /project/code/project/ && sh startup-jar.sh app"

Complete ci script (take the doctor side as an example)

image: java:8

stages:
  - package
  - build

services:
  - docker:dind

#At present, only the jar package is required, which depends on the host, and the cache is not required
#cache:
# key: push-maven
# paths:
#   - /root/.m2/
#   - target/*.jar

maven-package:
  image: maven:3.6.3-jdk-8
  stage: package
  script:
    - echo "===============directory structure  ===============" `ls`
    - echo "=============== Start compiling the source code, including unit tests, in target Directory generation jar file ==============="
    - mvn clean install
    - echo "target folder" `ls target/`
  artifacts:
    paths:
      - target/*.jar

build-deploy-docker-dev:
  image: docker:latest
  stage: build
  environment: dev
  only:
    - master
  before_script:
    - echo "=============== testing environment ==============="
    - echo "=====Change Ali image source====="
    - sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
    - echo "=====install ssh====="
    - which ssh-agent || ( apk update && apk --no-cache add openssh-client )
    - echo "=====to configure ssh file====="
    - mkdir -p ~/.ssh
    - cp ${SSH_PRIVATE_KEY}  ~/.ssh/id_rsa
    - chmod 700 ~/.ssh/id_rsa
    - cp ${SSH_CONFIG}  ~/.ssh/config
    - cat /etc/*-release
    - echo "=====install rsync====="
    - command -v rsync || (apk update && apk --no-cache add rsync)
  script:
    - echo "===== Recovered from cache target folder ====="
    - ls target/
    - echo "===== Replace the file name to facilitate subsequent release ====="
    - mv target/app*.jar target/app.jar
    - ssh -o StrictHostKeyChecking=no -T project-dev "cd /project/code/project/ && sh shutdown.sh app"
    - rsync -rve ssh target/app.jar root@project-dev:/project/code/project/app/app
    - ssh -o StrictHostKeyChecking=no -T project-dev "source /etc/profile && cd /project/code/project/ && sh startup-jar.sh app"

build-deploy-docker-pro:
  image: docker:latest
  stage: build
  environment: work
  only:
    - release
  script:
    - echo ""=============== production environment  "==============="
    - echo "Recovered from cache target folder" `ls target/`
    - echo "todo ..."

Problems encountered

Error reporting when using ssh connection to start project

The answer is given in this article: https://blog.csdn.net/nklinsirui/article/details/104673286/?utm_medium=distribute.pc_relevant.none -task-blog-baidujs_ title-1&spm=1001.2101.3001.4242

At the end of the paper, two solutions are given. It is more convenient to use method 2, so the execution command can be written as

ssh -o StrictHostKeyChecking=no -T project-dev "source /etc/profile && cd /project/code/project/ && bash startup.sh app"

The rsync command is undefined

The reason for this problem is: our gitlab runner uses alpine linux, which is a very compact version of linux system. There is no rsync tool installed, so it needs to be installed manually.

apk update && apk --no-cache add rsync

Due to network reasons, the direct installation fails. You need to replace the image source first. You can use Ali source

sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories

rsync error code 255

It looks like ssh authentication failed. This problem is also encountered on the Internet: https://bobcares.com/blog/rsync-error-code-255/

However, as mentioned in the installation document, this error is still reported.

I tried to replace the command order, put the ssh connection first, and ran successfully.