Troubleshooting Gitlab-runner Build Failures

Posted by aod on Fri, 17 May 2019 02:11:47 +0200

Troubleshooting Gitlab-runner build failures:

Causes of the problem:

Automated build based on gitlab ci. job runs with errors when merge_requests request is initiated to start building. It is officially recommended to upgrade gitlab-runner version. Build runs with errors after version upgrade. The errors are as follows:

Running with gitlab-runner 11.10.1 (1f513601)
 on jp.ptmind.com 0ea06a67
Using Shell executor...
Running on jp.ptmind.com...
Reinitialized existing Git repository in /data/builds/0ea06a67/0/. . . backend/.git/
Fetching changes with git depth set to 10...
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly
ERROR: Job failed: exit status 1

Investigation and resolution:

1: Upgrade the runner version to the latest version, clean up the old files directory of runner build directory during upgrade;

After the above configuration modifications, the first start build can succeed, and merge_requests can be executed again to trigger the build, or errors will occur, the problem remains unresolved.

build:common:
  stage: build
  tags:
    - python3
  script:
    - cp src/requirements_shared.txt src/common/requirements.txt
. . . . . . . . . . . 
  only:
    refs:
      - develop
      - staging
      - master
      - merge_requests
    changes:

      - src/common/**/*

2: suspect ci configuration logic issues, disassemble scripts, separate merge_requests from the listening branch;

build:common:
  stage: build
  tags:
    - python3
  script:
    - cp src/requirements_shared.txt src/common/requirements.txt
. . . . . . . . . . . 
  only:
  - merge_requests
    changes:
      - src/common/**/*

Execute the build again, and build fails occasionally. Click Retry to succeed...

3: There has been feedback from people looking for information that low git versions can lead to instability with new git features.The git version upgrade is performed below;

1.1 Delete the old git version of runner first;

Note that gitlab-runner will also be deleted at this time. It is recommended that the gitlab management platform delete runner now and add runner again after version upgrade.
The upgrade steps are as follows:

Delete old git
yum remove git -y

Install 3-party yum source, centos7 base warehouse, git version only available to 1.8.3, unable to use some new git 2 features

yum install https://centos7.iuscommunity.org/ius-release.rpm

Install a new version of git
yum install git2u -y

Verify Version
git --version

Reinstall runner
yum install gitlab-runner -y

Modify the runner configuration:

vi /etc/systemd/system/gitlab-runner.service

ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"

//Change to:

ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/data/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "root"
Re-register runner
gitlab-runner register

Start the runner service
systemctl daemon-reload
systemctl start gitlab-runner

###References:

https://gitlab.com/gitlab-org/gitlab-ce/issues/60466

Topics: Linux GitLab git yum shell