Troubleshooting Gitlab-runner build failures:
Causes of the problem:
Automated build based on gitlab ci. job runs with errors when merge_requests request is initiated to start building. It is officially recommended to upgrade gitlab-runner version. Build runs with errors after version upgrade. The errors are as follows:
Running with gitlab-runner 11.10.1 (1f513601) on jp.ptmind.com 0ea06a67 Using Shell executor... Running on jp.ptmind.com... Reinitialized existing Git repository in /data/builds/0ea06a67/0/. . . backend/.git/ Fetching changes with git depth set to 10... fatal: git fetch-pack: expected shallow list fatal: The remote end hung up unexpectedly ERROR: Job failed: exit status 1
Investigation and resolution:
1: Upgrade the runner version to the latest version, clean up the old files directory of runner build directory during upgrade;
After the above configuration modifications, the first start build can succeed, and merge_requests can be executed again to trigger the build, or errors will occur, the problem remains unresolved.
build:common: stage: build tags: - python3 script: - cp src/requirements_shared.txt src/common/requirements.txt . . . . . . . . . . . only: refs: - develop - staging - master - merge_requests changes: - src/common/**/*
2: suspect ci configuration logic issues, disassemble scripts, separate merge_requests from the listening branch;
build:common: stage: build tags: - python3 script: - cp src/requirements_shared.txt src/common/requirements.txt . . . . . . . . . . . only: - merge_requests changes: - src/common/**/*
Execute the build again, and build fails occasionally. Click Retry to succeed...
3: There has been feedback from people looking for information that low git versions can lead to instability with new git features.The git version upgrade is performed below;
1.1 Delete the old git version of runner first;
Note that gitlab-runner will also be deleted at this time. It is recommended that the gitlab management platform delete runner now and add runner again after version upgrade.
The upgrade steps are as follows:
Delete old git yum remove git -y Install 3-party yum source, centos7 base warehouse, git version only available to 1.8.3, unable to use some new git 2 features yum install https://centos7.iuscommunity.org/ius-release.rpm Install a new version of git yum install git2u -y Verify Version git --version Reinstall runner yum install gitlab-runner -y
Modify the runner configuration:
vi /etc/systemd/system/gitlab-runner.service ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner" //Change to: ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/data/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "root"
Re-register runner gitlab-runner register Start the runner service systemctl daemon-reload systemctl start gitlab-runner
###References: