docker deployment specification - integration with gitlab-ci

This specification is summarized as follows, based on a number of project practices

Continuously integrated project directory structure

A complete product will contain multiple services, such as web services, java services, python services, etc.
And the code for each service corresponds to a different code project, with the project directory structure shown below

[Product Name]-web/             #[web Front End Project]-->Front End Developers
  src/
  .gitlab-ci.yml     #Trigger CI action after submitting code
  build.sh           #Auxiliary script to build docker image, compile source code, build, tag, push, etc.
  nginx.conf         #Mirrors built on nginx, predefined configurations for nginx
  Dockerfile         
[Product Name]-java/            #[java engineering]-->java developers
  src/
  .gitlab-ci.yml   
  build.sh   
  Dockerfile 
[Product Name]-python/          #[python project]-->python developers
  src/
  .gitlab-ci.yml   
  pre.Dockerfile     #python's package installation takes time to build a pre image
  Dockerfile         #Every time code is modified, based on pre image, fast
  build.sh   
[Product Name]-deploy           #[Deployment Project]-->Architect, Operations and Maintenance Personnel
  conf
    docker-compose.yml #Scheduling scripts
  init-data  #Initialized data, such as database scripts, initial data at program load, etc.
  run-data   #Data during run, such as database storage directory, temporary file directory for program
  log        #Program Log Target
  back       #Program Backup Directory

Build a mirror build.sh

Through build. The sh script builds the mirror and executes the command build.sh Project Name Branch Name

• Read project and branch names
• Build a project from source code
• Build a mirror in the following format: Storage Address/Project Name: Branch Name
• push mirror to warehouse server

For the second step, build the project from source code, using different technology stacks. SH writings vary, sometimes slightly different for the same technology stack

# Read project and branch names
if [ -n "$1" ] ; then reposityName=$1; else echo -e "usage:\n\t./build.sh reposityName [tagName]" && exit 1 ;  fi
if [ -n "$2" ] ; then tagName=$2; else tagName="master" ; fi
imageName=$reposityName:$tagName

echo "===>step1: build web for $imageName"
npm install
if [ -d dist ]; then rm -rf dist; fi
npm run build:prod
echo "===>step2: build docker image:$imageName"
docker build -t $imageName .
imageName2=10.1.192.120:5000/$imageName
docker tag $imageName $imageName2
echo "===>step3: push docker image to reposity"
docker push  $imageName2

Dockerfile description

The corresponding Dockerfile s for different technology stacks are different

Dockerfile for web Services

FROM nginx
# Create app directory
RUN mkdir -p /usr/src/app
# Copy Deployment File
COPY dist  /usr/src/app/
COPY nginx.conf  /etc/nginx/

Dockerfile for java services

FROM java:8

WORKDIR /usr/local/app
# Copy the jar into the mirror, corresponding to different items, where the jar package names are different
COPY ./scientific-system/target/scientific-system-2.6.jar  /usr/local/app/
## Wait gadget, wait for dependent services to be ready
COPY ./wait /wait  
RUN chmod +x /wait
CMD /wait &&  java -jar scientific-system-2.6.jar

Dockerfile for python service

Because python takes a long time to start packaging with a header, it is divided into two steps.

• Pre.Dockerfile is pre-built
• Trigger Dockerfile to build when project source code changes

Pre. An example of a Dockerfile is as follows:

FROM centos7-python:3.8.10
COPY requirements.txt .
RUN pip install  -r requirements.txt

Manually execute docker build-f pre when a dependent package needs to be installed. Dockerfile-t product name: pre.. Build an intermediate image first, and when the source code is submitted, the corresponding Dockerfile is as follows:

FROM Product Name:pre
WORKDIR /usr/local/app
COPY .  /usr/local/app
ENTRYPOINT ["python"]
CMD ["-u","main_api.py"]

Integrated with CI. gitlab-ci.yml

• Calculate project catalog prefix: Rule product name_ deploy_ Branch name
• Build Project:
  • Enter Source Directory
  • Switch branches and pull the latest code
  • Build the mirror and upload it to the mirror warehouse server in the format: warehouse server/project name: branch name
• Update Cluster Services:
  • Enter Deployment Directory
  • Pull Mirror, Mirror Name Format: Warehouse Server/Project Name: Branch Name
  • Restart Service

stages:
  - deploy
deploy:
  stage: deploy
  tags:
    - ai-test-by-shell
  script:
    # Calculating the prefix of a project catalog can either extract the first two words automatically or write them to death manually, such as export PROJECT_DEPLOY=ywz_deploy
    - export PROJECT_DEPLOY=$(echo $CI_PROJECT_NAME |awk -F '_' '{print $1"_"$2"_deploy"}')
    - export PROJECT_DEPLOY="$PROJECT_DEPLOY_$CI_COMMIT_BRANCH"  # Combining with branches, ywz_deploy_master
    - echo "CI_PROJECT_NAME=$CI_PROJECT_NAME   CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH PROJECT_DEPLOY=$PROJECT_DEPLOY"
    # Build Project
    - source /root/.bashrc && cd /home/build/$CI_PROJECT_NAME  && git checkout $CI_COMMIT_BRANCH && git pull && sh build.sh $CI_PROJECT_NAME $CI_COMMIT_BRANCH
    # Update Cluster  
    - ssh root@w118 "source /root/.bashrc && cd /home/build/${PROJECT_DEPLOY}/conf && docker pull 10.1.192.120:5000/$CI_PROJECT_NAME:$CI_COMMIT_BRANCH && docker-compose up -d --build"
  only:
    - master

Reference resources

• How to handle variables with underlines behind them