preface

The story happened on a dark and windy night. An unusual phone call came and said that the business was in a hurry to go online, but their API package could not be uploaded to the company's maven private library. The leader asked me to support and see how to solve it. After years of unreliable intuition, the disk should be full. So he knocked it down neatly

df -lh

Sure enough, the disk is full, and / var/lib/docker/overlay basically fills the disk. Then enter

docker system df

View the disk size occupied by docker. When thinking about whether to apply for the process sheet to call the operation and maintenance to expand the disk or manually clean the disk, the phone came again and said that the problem had not been found. Can you solve it quickly. After answering the phone, I was inexplicably upset, so I knocked down the following order

docker system prune

This command can be used to clean up disks, delete closed containers, useless data volumes and networks, and dangling images (i.e. images without tag s). Then a phone call came again and said that gitlab could not be accessed. The answer I gave at that time was that the disk was full and gitlab should have stopped. I'll restart the gitlab container later. When I intend to restart gitlab, I'll knock the command

docker ps -a

I want to fish the gitlab container, and then I'm finished. Docker PS-A can't see any container. Because the container of gitlab was installed by the former architect, I didn't know what form it was installed at that time, so I fed this problem back to the leader. Through the leader, I got the docker-compose.yml that started gitlab at that time. The example is as follows

version: '2'

services:
  redis:
    restart: always
    container_name: gitlab_redis
    image: sameersbn/redis:latest
    command:
    - --loglevel warning
    volumes:
    - /usr/local/docker/gitlab/redis:/var/lib/redis:Z
    network_mode: bridge

  postgresql:
    restart: always
    container_name: gitlab_postgresql
    image: sameersbn/postgresql:9.6-2
    volumes:
    - /usr/local/docker/gitlab/postgresql:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm
    network_mode: bridge

  gitlab:
    restart: always
    container_name: gitlab
    image: sameersbn/gitlab:10.4.2-1
    links:
    - redis
    - postgresql
    network_mode: bridge
    ports:
    - "10080:80"
    - "10022:22"
    volumes:
    - /usr/local/docker/gitlab/gitlab:/home/git/data:Z
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Asia/Shanghai
    - GITLAB_TIMEZONE=Beijing

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=gitlab.common.com
    - GITLAB_PORT=
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=

After looking at the files, I saw that the file directory was mounted, and then I went to the mounted file directory to see if the files were there. Fortunately, the files were all there, so I was relieved to knock them down

docker-compose -f gitlab.yml up -d

As soon as this order was knocked down, the road of recovery opened magnificently

text

When I knocked down the command and saw that the container started normally and planned to continue cleaning the disk, suddenly wechat received messages from several developers saying that their gitlab login showed that the user or password was invalid, so I also used my account. My account is a manager account. Ha ha, it's a bastard, Input my familiar user name and password on the interface, and unexpectedly prompt me that my user name or password is invalid. I feel a little flustered. I feel that I have done a great thing.

Things are a little beyond my understanding. In my opinion, if you do mount, there will be no data loss normally. So you find a friend for emergency assistance and communicate with him later, that is, restore with backup data first, if there is a backup. Then I saw the backups directory and the tar in the directory where gitlab was installed. Suddenly, I felt like I had found a treasure, but when the treasure was found, where was the key to the treasure, and how could I recover the backup data. Because I didn't know about sameersbn, I went to their github. Their github link is as follows

https://github.com/sameersbn/docker-gitlab

Find the following introduction through their github

When using docker-compose you may use the following command to execute the restore.

docker-compose run --rm gitlab app:rake gitlab:backup:restore # List available backups
docker-compose run --rm gitlab app:rake gitlab:backup:restore BACKUP=1515629493_2020_12_06_13.10.0 # Choose to restore from 1515629493

Follow the orders and knock down the following commands

docker-compose -f gitlab.yml run --rm gitlab app:rake gitlab:backup:restore

After about a song, I opened the gitlab login interface again. With a nervous mood, I entered my user name and password. I was pleasantly surprised to find that the login was successful. Look at the warehouse. Everything is still there

summary

I've talked a lot. In fact, I've summed up a few words, that is, backup, backup, say important things three times. Secondly, I'm doing some commands that may delete data or change data. Be careful, be careful, and be careful again. Finally, if Google and Baidu can't find the answer, remember to go to the official website or their github website to find the answer, such as sameersbn gitlab link
https://github.com/sameersbn/docker-gitlab