Meet Docker

definition

• Docker is an open-source application container engine based on Go language. Based on cgroup, namespace of Linux kernel and Union FS of OverlayFS class, it encapsulates and isolates processes, which belongs to virtualization technology at the operating system level.

• Docker allows developers to package their applications and dependencies into a lightweight, portable container, and then publish them to any popular Linux machine, which can also realize virtualization;

• Containers are completely sandboxed and have no interface with each other.

framework

• Image: the OS is divided into kernel and user space. The user space of Linux is the root file system. Docker image is equivalent to a root file system. Using the technology of Union FS, it is designed into a multi-layer architecture. Each layer can be reused, so as to reduce the pulling of image or the volume of image.

• Container: similar to the relationship between classes and objects in object-oriented, a container can be understood as a mirror instance and can be created, started, stopped, deleted, paused, etc. The process in the container runs in an isolated environment. If the container needs to write, it should use data volume. If it writes data directly to the container, the data will disappear after the container disappears, and the data volume will persist in the host machine.

• Warehouse: Docker Registry is the place where image warehouses are stored. Each image warehouse can have multiple labels, and each label corresponds to an image, which can be used to identify the image version.

  The Docker Registry public service can provide users with downloads of public images and upload their own images to individual users of the public service. Common public services include Docker Hub, Netease cloud image service, DaoCloud image market, Alibaba cloud image library, etc.

  We can pull the image from the shared Docker Registry by using the docker pull command. Before pulling, it is recommended to read the operating instructions of the image, such as version number, demo, etc. the website is:

  https://hub.docker.com/

  We can also set up a private Docker Registry on our own server to manage our image files locally.

advantage

• Simplified configuration
• Code Pipeline management
• Isolated applications
• Consolidate servers to make more efficient use of resources
• Commissioning capacity
• Rapid deployment

Docker installation

Environmental Science

OS: ubuntu18.04
Docker: 19.03.8

Uninstall the old version of Docker on the machine (skip if not installed before)

1. Delete docker component

sudo apt-get remove docker docker-ce docker-engine docker.io containerd runc

2. Delete residual images, containers, data volumes, networks, configurations, etc

sudo apt-get purge docker-ce

sudo rm -rf /var/lib/docker

APT installation (recommended)

1. Update APT package index

sudo apt-get update

2. Add packages and CA certificates using HTTPS transport

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

3. Add GPG key for software source

# Aliyuan
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

# Official source
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

4. Set the stable repository to source.list Add Docker software source to

# Ali
sudo add-apt-repository \
"deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu \
 $(lsb_release -cs) stable"

# official
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

5. View the installed Docker version

apt-cache madison docker-ce

6. Installing the Dodger engine

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Other installation methods

1. deb installation

get into https://download.docker.com/linux/ubuntu/dists/ Select Ubuntu version, then go to pool/stable /, select amd64, armhf, arm64, ppc64el, or s390x, download the. deb file, and then execute the following command to install

sudo dpkg -i /path/to/package.deb

2. Easy script installation

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Verify installation

Enter the docker version, and the following information will appear, and the installation is successful. At the same time, you can see that the docker version of this installation is 19.03.8. You can also see the API version and the GO version. The following small line reminds you that the current permission is not enough, which can be solved through the Docker user group in the next section.

docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/version: dial unix /var/run/docker.sock: connect: permission denied

Set up Docker user group

By default, the docker command uses Unix socket to communicate with the docker engine. Only root and docker group users can access the Unix socket of docker engine. For security reasons, root is not directly used on Linux systems. Therefore, it is better to add users who need to use docker to the docker user group;

Restart the terminal after executing the following command.

sudo groupadd docker

sudo usermod -aG docker Current user

Configuration and startup of Docker

Create and modify daemon.json file

Take the configuration of domestic image sources as an example. Because the image service may be down, it is recommended to configure multiple image sources at the same time. Here, the image sources of Alibaba or daocloud are preferred.

Alibaba image address: https://cr.console.aliyun.com/

cd /etc/docker
sudo touch daemon.json

sudo vim daemon.json
# Add the following
{
  "registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"]
}

esc -> :wq -> enter

Start Docker

# Set to start Docker service automatically after power on
sudo systemctl enable docker

# Start Docker service
sudo systemctl start docker

After modification, you need to reload the configuration and restart the Docker service

sudo systemctl daemon-reload
sudo systemctl restart docker

See if changes take effect

sudo systemctl status docker -l
sudo docker info

Start a mirror

docker run hello-world
# Output display:
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete 
Digest: sha256:8e3114318a995a1ee497790535e7b88365222a21771ae7e53687ad76563e8e76
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

Build private warehouse

In this section, we use the official docker registry tool to build a private warehouse for storing our images. When we use it, we only need to pull it from the local area to save time.

Install docker registry

1. View the registry image in the image source

docker search registry

2. Pull image

This step can be omitted. You can directly execute the run command to start the container, which will be pulled automatically when you specify the image.

# This instruction will pull the official latest label of the image source by default
docker pull registry

# View local image
docker images

3. Run container

docker run -d -p 5000:5000 --name mda_registry \
-v /usr/mda_registry:/var/lib/registry \
--restart=always registry:latest

# View started containers
docker ps

# View container MDA_ IP address of registry (172.17.0.2)
docker inspect mda_registry| grep IPAddress

-d: Background run container
-p: Map host ports to container ports
– Name: container name
-v map the host directory to the in container directory
– restart: container restart mechanism
registry:latest : image used and image label

4. View image of private warehouse

curl http://172.17.0.2:5000/v2/_catalog

5. Upload an image (take python image for example)

Pull a python version 3.8 image from docker hub. It is recommended to log in to docker hub to view the image label and use samples before pulling, https://hub.docker.com/_/python?tab=tags

# Pull an image from Docker Hub
docker pull python:3.8
# see
docker images

Label in the form of:

IP address of private warehouse: port number / image name

docker tag python:3.8 172.17.0.2:5000/python:3.8
# docker images, one more image

When uploading the Docker image through the Docker Registry tool, it can only be done through https, so you need to upload the Docker image in / etc/docker/daemon.json Join:

sudo vim /etc/docker/daemon.json

{
	  "registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"],
	  "insecure-registries":["172.17.0.2:5000"],
}

# esc -> :wq -> enter
sudo systemctl daemon-reload
sudo systemctl restart docker
docker ps

Reload the configuration file and restart the docker service. Since - restart=always is set, the docker will restart, and the register can be viewed through the docker ps command.

Upload the image to the private warehouse through the push command, and view

docker push 172.17.0.2:5000/python:3.8

curl http://172.17.0.2:5000/v2/_catalog

6. Use image of local warehouse

First, delete the python image pulled from the official database through rmi image ID (or image name: tag), and view it through docker images, then pull the image from the local warehouse, and view:

docker pull 172.17.0.2:5000/python:3.8
docker images

7. Other operations

• View images in the warehouse

curl http://172.17.0.2:5000/v2/_catalog

• View a list of versions of a mirror

curl  http://172.17.0.2:5000/v2/python/tags/list

• View configuration of private warehouse

docker exec -it  registry sh -c 'cat /etc/docker/registry/config.yml'

• Delete image in private warehouse
  ？？？

Make a mirror image

Mirrored structure

The principle of image is to use the technology of Union FS to design it as a hierarchical storage architecture, which is composed of multi-layer file system.

Build layer by layer, the former layer is the foundation of the latter layer.

• Convenient extension: when making images through dockerfile, try to put the invariable ones in front, such as environment installation, environment variable setting, etc., and put the steps that may change behind, such as code upload, starting execution command, etc;
• Shared resources: for example, we make two images based on Ubuntu 18.04, one java image and one python image, which can share Ubuntu 18.04 image layer.

Two ways of making

Take creating a java image and uploading code to the image as an example:

• Start a container with ubuntu as the image through the run command, then enter the running container to install the java environment, configure the environment variables, upload the local code to the container through the docker cp command, and finally create the container as a new image through the docker commit command.
• Write a Dockerfile to create a new image.

The Dockerfile mode is recommended:

• The image generated by commit mode is a kind of image after black box operation, which is difficult to describe the specific information of the image, so it is difficult to maintain, expand and reuse;
• If the image generated by comiit mode is further expanded, the previous image cannot be changed, resulting in the image becoming more and more bloated.

Making a image with Java 1.8 environment based on Ubuntu 18.04 image

cd /usr/dockerfiles
sudo vim dockerfiles

### dockerfile
FROM 120.133.17.174:5000/ubuntu:18.04

RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone

ADD jdk-8u144-linux-x64.tar.gz /usr/local/java/
ENV JAVA_HOME /usr/local/java/jdk1.8.0_144
ENV JRE_HOME $JAVA_HOME/jre
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
ENV PATH $JAVA_HOME/bin:$PATH
COPY ./VideoSearch/  /usr/VideoExtract/VideoSearch/
COPY ./lusi_workspace/  /usr/VideoExtract/lusi_workspace/
ENV LANG C.UTF-8
RUN cp  /usr/VideoExtract/VideoSearch/videosearch.conf /etc/ld.so.conf.d/ && ldconfig

• FROM: it must be the first instruction in the dockerfile, which is used to specify a basic image and customize it later. The basic image can be mysql, redis and other service images, java, python and other language images, ubuntu, centos and other system images, or empty image scratch , used to run services that do not need any environment support, such as go language developed microservices. If the image warehouse is not specified, the official image warehouse will be used by default, or the image warehouse I have built can be used as above;
• RUN: execute the command, because each RUN of RUN will result in another layer of image, so it is recommended to merge multiple RUN, and use & & to connect;
• COPY: the files of the host are copied to the image. Wildcards can also be used for the source path, such as COPY home *, COPY home?. TXT. All metadata of the source file will be retained, such as read-write modification permission and file modification time;
• ADD: similar to COPY function, but when copying compressed files, it will automatically decompress to the image and delete the compressed package;
• CMD: the command executed when the container is started, which can also be specified in the docker run command;
• ENV: set environment variable, you can use ENV KEY VALUE or ENV KEY=VALUE;
• VOLUME: to create one or more mount points, VOLUME /data or VOLUME ["/data1","/data2"], the instruction will mount the / data directory in the container and a path of the host machine, but it is not allowed to mount to the path of the specified host machine here. To specify the path, you need to specify it when the docker run starts the container, and this instruction will be overwritten. You can view the mount status through the dokcer inspect ion container name (container ID);
• EXPOSE: declare the port to be exposed, just declare it. In order for the user to understand the image, the actual effect needs to be specified through the p parameter when docker run;
• WORKDIR: Specifies the working path to prepare for the next command. You cannot RUN cd /data in the dockerfile to execute the command under data, so you need WORKDIR /data to select;
• USER: specify the current USER
• HEALTHCHECK: set the command to check the health status of the container, including the interval, timeout and cmd parameters. You can also use HEALTHCHECK NONE to shield the health check command set by the basic image. You can view the health status in docker ps;
• On build: when the basic image is from another dockerfile, it will execute.

PS: it is recommended to add the following instructions to each Dockerfile to ensure that the container's time zone is China's time zone, otherwise it will be 8 hours away from China's time zone.

RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone

Common demo for making image

1. Debian replace apt get source

The Linux version of the official Python 3.8 image trial is Debian, so when installing apt, you need to change the apt source to the domestic Ali source:

# Create a new file in the directory of dockerfile sources.list  Files, writing
deb http://mirrors.aliyun.com/debian buster main
deb http://mirrors.aliyun.com/debian buster-updates main

# Write in dockerfile
RUN cp ./sources.list /etc/apt
RUN  apt-get clean -y && apt-get update -y

Is the code inside or outside the Docker

• In dockerfile, the pull command is executed through RUN or CMD instruction to connect git code management;
• Directly execute COPY in the dockerfile and upload the code to the image;
• Mount to local;

tips for making images

• If the image is complex, try to split it into basic image (system) and application image (program). For example, if you want to make a crawler image, you can first make a python image, and then make a crawler application image based on the python image;
• When building an image, if it has the same layer as other images, the cache will be used directly (it will be compared with all the sub images generated by the basic image), so try to put the layer that does not have long changes in front;
• A new layer will be built during RUN, so it is recommended to use & & to connect multiple RUN commands, and use backslash to divide them into multiple lines;
• The recommended way to install apt get is:

  RUN apt-get update && \
  apt-get install -y package && \
  rm -rf /var/lib/apt/lists/*
  

• ENV will also generate an intermediate layer, so it can be integrated into RUN

  RUN export ADMIN_USER="mark" \
  && echo $ADMIN_USER > ./mark \
  && unset ADMIN_USER
  

• In case of COPY or ADD, both the source path and the target path will be provided with a backslash "\"

Execute build instructions

docker build -t java:8 -f /usr/dockerfiles/dockerfile .

-t: Specify the name of the image: label;

-f: Specify the path to use dockerfile;

The last point: Specifies the build image context to be used by the dockerfile, which is simply the root path of the COPY command. The. / of the source path parameter in COPY is the path specified by the build command point, which can also be written as an absolute path;

In addition, you can also specify some parameters, such as CPU usage weight, quota, cycle, memory maximum, swap maximum, ulimit, netwokd mode, etc.

Other operations

• delete mirror

  docker rmi image name
  

• View image creation history

  docker history image name
  

• Import and export

  docker save -o filename.tar Image name
  
  docker load -i file name
  
  # Using container file of docker export to generate image
  docker import  filename.tar  818/ubuntu:1804 
  

Run container

Start container

Start the image we built above with the following command

# docker run parameter mirror CMD
docker run -itd --name="extracvideo" java:8  java -jar testexe.jar

Parameters:
-d: Start the container in the background and return the container ID
-i: Interactive mode running container, used with - t
-t: Assign a pseudo terminal to the container
-p: Port mapping, host port: container port
– Name: container name
-e: Environment variable
– env file: read environment variables from the specified file
-m: Maximum memory used by container
-v: Bind a volume, host path: in container path

Here, the parameter - itd is added at startup, because if no command is added at startup, or the command is executed, the container will stop running immediately. After the parameter - itd is added, even if no command is entered, the container will not exit.

When docker run is executed, first check whether the specified image exists locally. If it does not exist, it will be pull ed to the shared warehouse. After startup, a read-write layer will be mounted outside the read-only image layer, and then execute the specified command. After execution, the container will be terminated. When the container is deleted, the read-write layer will not be preserved, so when reading and writing to the container, if you want to persist the content, you need to make a mount with the host machine.

Start a container. If an error is reported as follows:

docker: Error response from daemon: endpoint with name extractvideo_90 already exists in network bridge.

It is possible that the occupation of the network remains in the docker service when deleting the container. You need to clear it manually:

docker network inspect bridge
# Find the container occupied by the network from the Containers_ name
docker network disconnect --force bridge container_name

Other operations

• View container details

  Docker inspection container ID
  

• View process information in the container

  docker top container ID
  

• View container log

  docker logs container ID
  

• Replication inside and outside the container

  docker cp host path container ID: path in container
  
  docker cp container ID: path in container host path   
  

• Enter container

  docker exec -it container ID /bin/bash
  
  # Back to the host machine
  ctrl + p + q
  

• Terminate container

  docker stop container ID
  

• Delete container

  # According to ID
  docker rm container ID
  
  # Force delete (running)
  docker rm -f container ID
  
  # Clean up stopped containers
  docker container prune
  

• Import and export

  # export
  docker export -o filename.tar container ID
  # Import container file as image 
  docker import  filename.tar  818/ubuntu:1804 
  

• View the Linux version of the container

  You can only use the cat /etc/issue command. If you use uname -a or cat /proc/version, you will see the version of the host machine

  docker exec -it container id /bin/bash
  cat /etc/issue
  

PS: we'd better check the IP and system time of the container to avoid other errors when running the service.

Docker network

reference resources: https://www.jianshu.com/p/bf8eb25d180e

Host mode (shared network of container and host machine)

After adding the -- network host parameter to the docker run command, the starting container tries the host network mode, which has the following characteristics:

• The container and the host are under the same network namespace and use the same network protocol stack. The container can directly use all network interfaces of the host

• The host mode cannot use port mapping and custom routing rules, which are consistent with the host. The - p and - icc parameters are invalid

• The network model under the host mode is the simplest and the lowest latency mode. The container process communicates directly with the host network interface, which is consistent with the physical machine performance

Container's log

Two kinds of logs

The Docker logs are divided into the logs generated by the container engine and the logs generated by the services in the container. The logs generated by the container engine can be viewed by the following command:

journalctl -u docker

Service log

• see

  docker logs container ID (name)
  #Trace output
   docker logs -f container ID (name)
  #Show timestamps
   Docker logs - tcontainer ID (name)
  
  View the log file through the cat or tail command. The path is:
  /var/lib/docker/containers / container ID / container ID-json.log
  

• Log drive
  There are JSON file (default), loca, syslog, journal D. you can view the total set log driver and the log driver of a container through docker info and docker inspect ion container ID;

  You can daemon.json , where:

  Max size is the maximum volume of a single log file, and max file is the maximum number of log files produced by a container

  vim /etc/docker/daemon.json
  {
    "log-driver": "json-file",
    "log-opts": {
      "max-size": "10m",
      "max-file":"1",
    }
  }
  
  #systemctl daemon-reload
  #systemctl restart docker
  

  You can also specify the log driver of a container in the docker run command:

  docker run -itd --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 java8
  

Container common commands summary

• Container instance correlation: docker [run | start | stop | restart | kill | RM | attach | exec | pause | unpause | CP]

• Container view information: docker [stats | PS | inspect | top | logs | wait | export | port | diff]

• Image correlation - docker [images | RMI | tag | build | commit | history | save | import]

• Image warehouse related: docker [login | pull | push | search]

• Container service related: docker [info | version | events]

docker stats # View the CPU, memory, IO and other resources occupied by each container
docker inspect # Go deep inside the container to get all the information of the container
docker logs # View the container's log (stdout/stderr)
docker events # Get real-time events of docker server
docker port # Show port mapping for container
docker top # Display the process information of the container
docker diff # Show changes before and after the container file system
docker history # Display the historical command to generate a mirror

docker create # Create a container but do not start it
docker run # Create and start a container
docker stop # Stop the container and send SIGTERM
docker start # Start a container in a stopped state
docker restart # Restart a container
docker rm # Delete a container
docker kill # Send signal to container, default SIGKILL
docker attach # Connect (enter) to a running container
docker wait # Block a container until it stops

docker ps # Display containers with Up status
docker ps -a # Show all containers, including Up and exited


docker cp # Copy files or directories from the container
docker export # Export the entire file system of the container as a tar package without layers, tag s and other information

docker exec # Execute a command in the container, and bash can be executed to enter the interaction.

docker images # Display a list of all local mirrors
docker import # Create an image from a tar package, often in combination with export
docker build # Use Dockerfile to create image (recommended)
docker commit # Create a mirror from a container
docker rmi # Delete a mirror
docker load # Create a mirror from a tar package and use it with save
docker save # Save an image as a tar package with layers and tag information
docker tag # Create an alias for the image