Requirements: it is used to provide project alternatives with high stability requirements for cicd deployment.
Background: Although Jenkins is superior to codebuild in terms of the process of building cicd by itself, Jenkins will have occasional downtime, resulting in the problem that a large number of services cannot be published. Therefore, it is necessary to have a server free construction method with high stability like codebuild.
Overall architecture
As shown in the figure, a gitlab configured with a corresponding webhook has a push event, which will trigger the gitpull function in lambda to pull the project information of the project. The whole cicd process uses the key to transmit data packets for security. All encryption behaviors are that another function in lambda, createshkey, requests the kms service to create and read the key, Then gitpull sends the pulled git code and all the information of the modified project warehouse (including branches, project names, etc.) to codebuild to build the project. Finally, codebuild pulls the source code of the project, pushes the constructed image to the ecr image warehouse, and operates eks, Publish the latest version of the image in the image warehouse to the corresponding deployment in the cluster.
Create process
AWS officially provides a demo, a cloud formation (AWS service stack) that can synchronize gitlab to s3 bucket and package the source code.
Refer to the official documents for the specific creation process: https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-simple-s3.html
During the creation process, pay attention to modifying the availability zone.
After the entire stack is created, modify lambda and codebuild in a customized way. First, the existing condition is that the push event of gitlab library configured with webhook will trigger the operation of lambda. Lambda will send the project information to codebuild, which will use the information to pull the correct branch code for construction. In this analysis, in fact, what needs to be modified most is the content of codebuild. The modification of lambda can be customized according to the needs of later project construction.
The following is the content of codebuild spec
version: 0.2
env:
exported-variables:
- GIT_COMMIT_ID
- GIT_COMMIT_MSG
phases:
install:
runtime-versions:
python: 3.7
# commands:
# - pip3 install boto3
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
build:
commands:
- echo "=======================Start-gitpull============================="
- echo "Getting the SSH Private Key"
- |
python3 - << "EOF"
from boto3 import client
import os
s3 = client('s3')
kms = client('kms')
enckey = s3.get_object(Bucket=os.getenv('KeyBucket'), Key=os.getenv('KeyObject'))['Body'].read()
privkey = kms.decrypt(CiphertextBlob=enckey)['Plaintext']
with open('enc_key.pem', 'w') as f:
print(privkey.decode("utf-8"), file=f)
EOF
- mv ./enc_key.pem ~/.ssh/id_rsa
- ls ~/.ssh/
- echo "Setting SSH config profile"
- |
cat > ~/.ssh/config <<EOF
Host *
AddKeysToAgent yes
StrictHostKeyChecking no
IdentityFile ~/.ssh/id_rsa
EOF
- chmod 600 ~/.ssh/id_rsa
- echo "Cloning the repository $GitUrl on branch $Branch"
- git clone --single-branch --depth=1 --branch $Branch $GitUrl .
- ls -alh
- export projectname=$(echo $outputbucketpath | tr '/' ' ' | awk '{print $2}')
- |
if [ "$Branch" = "deploy-staging" ]; then
export repo_name=$projectname"-staging"
export env="stage"
elif [ "$Branch" = "deploy-production" ]; then
export repo_name=$projectname"-production"
export env="prod"
else
return 1
fi
- |
if [ "$projectname" = "export" ]; then
export contextname="xxxx-bussiness"
elif [ "$Branch" = "deploy-staging" ]; then
export contextname="xxxx-bussiness-non-prod"
elif [ "$Branch" = "deploy-profuction" ]; then
export contextname="xxx-bussiness"
else
return 1
fi
- export GIT_COMMIT_ID=$(git rev-parse --short HEAD)
- echo $GIT_COMMIT_ID
- export GIT_COMMIT_MSG="$(git log -1 --pretty=%B)"
- echo $GIT_COMMIT_MSG
- pwd
- aws s3 cp s3:/xxx/xxx/settings.xml /opt/maven/conf/
- echo "=======================End-gitgull============================="
- echo "=======================Start-Build============================="
- |
export isjava=$(find ./ -name pom.xml)
if [ "$isjava" = "" ]; then
echo "it's node project"
echo "Start Npm install......................................"
npm install
echo "Start Npm run build...................................."
npm run build:$env
else
echo "it's java project"
mvn clean package -Dmaven.test.skip=true
fi
- echo "=======================End-Build============================="
- echo Build started on `date`
- echo Building the Docker image...
- echo "=====================Start-Docker-build======================"
- docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
- docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- echo "=========================End-Docker-build======================"
- docker image ls
- mkdir ~/.aws
- mkdir ~/.kube
- export imageurl=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- echo $imageurl
- aws s3 cp s3://xxx/xxx/kube/ ~/.kube/ --recursive
- aws s3 cp s3://xxx/xxx/awscli/ ~/.aws/ --recursive
- aws s3 cp s3://xxx/xxx/deploy.sh ./
- aws s3 cp s3://xxx/xxx/image.json ./
- sed -i 's#imageurl#'$imageurl'#g' image.json
- sed -i 's#deploymentname#'$deploymentname'#g' image.json
- ls -alh ~/
- chmod +x deploy.sh
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker image...
- docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$repo_name:$deploymentname_$GIT_COMMIT_ID
- kubectl config get-contexts
- echo $contextname
- ./deploy.sh $contextname $deploymentname $repo_name
Problem solving:
1. The permissions of codebuild on ecr, eks and other resources need to be allocated first.
2. How to judge whether the project is a node project or a java project during the construction process.
3. Determine the publishing environment according to different branches.
4. If the maven configuration file involves a custom maven warehouse address, you can copy the configuration file from s3 to the build environment.
5. The configuration files of kubectl and aws are copied locally through s3.
6. The kubectl patch step is documented and scripted to solve the limitation of codebuild spec on cli in this step.
deploy.sh
#!/bin/bash kubectl --context $1 patch deployment $2 --patch "$(cat image.json)" --namespace $3
image.json
{
"spec": {
"template": {
"spec": {
"containers": [{
"name": "deploymentname",
"image": "imageurl"
}]
}
}
}
}
gitlab permission configuration
Use the PublicSSHKey output by cloudformation to create an ssh key for authentication.
lambda code content
from boto3 import client
import os
import time
import stat
import shutil
from ipaddress import ip_network, ip_address
import logging
import hmac
import hashlib
import distutils.util
exclude_git = bool(distutils.util.strtobool(os.environ['ExcludeGit']))
cleanup = False
key = 'enc_key'
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers[0].setFormatter(logging.Formatter('[%(asctime)s][%(levelname)s] %(message)s'))
logging.getLogger('boto3').setLevel(logging.ERROR)
logging.getLogger('botocore').setLevel(logging.ERROR)
s3 = client('s3')
kms = client('kms')
def lambda_handler(event, context):
print(event)
keybucket = event['context']['key-bucket']
outputbucket = event['context']['output-bucket']
pubkey = event['context']['public-key']
# Source IP ranges to allow requests from, if the IP is in one of these the request will not be checked for an api key
ipranges = []
if event['context']['allowed-ips']:
for i in event['context']['allowed-ips'].split(','):
ipranges.append(ip_network(u'%s' % i))
# APIKeys, it is recommended to use a different API key for each repo that uses this function
apikeys = event['context']['api-secrets'].split(',')
ip = ip_address(event['context']['source-ip'])
secure = False
if ipranges:
for net in ipranges:
if ip in net:
secure = True
if 'X-Git-Token' in event['params']['header'].keys():
print(event['params']['header']['X-Git-Token'])
if event['params']['header']['X-Git-Token'] in apikeys:
secure = True
if 'X-Gitlab-Token' in event['params']['header'].keys():
if event['params']['header']['X-Gitlab-Token'] in apikeys:
secure = True
if 'X-Hub-Signature' in event['params']['header'].keys():
for k in apikeys:
if 'use-sha256' in event['context']:
k1 = hmac.new(str(k).encode('utf-8'), str(event['context']['raw-body']).encode('utf-8'),
hashlib.sha256).hexdigest()
k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha256=', ''))
else:
k1 = hmac.new(str(k).encode('utf-8'), str(event['context']['raw-body']).encode('utf-8'),
hashlib.sha1).hexdigest()
k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha1=', ''))
if k1 == k2:
secure = True
# TODO: Add the ability to clone TFS repo using SSH keys
try:
# GitHub
full_name = event['body-json']['repository']['full_name']
except KeyError:
try:
# BitBucket #14
full_name = event['body-json']['repository']['fullName']
except KeyError:
try:
# GitLab
full_name = event['body-json']['repository']['path_with_namespace']
except KeyError:
try:
# GitLab 8.5+
full_name = event['body-json']['project']['path_with_namespace']
except KeyError:
try:
# BitBucket server
full_name = event['body-json']['repository']['name']
except KeyError:
# BitBucket pull-request
full_name = event['body-json']['pullRequest']['fromRef']['repository']['name']
if not secure:
logger.error('Source IP %s is not allowed' % event['context']['source-ip'])
raise Exception('Source IP %s is not allowed' % event['context']['source-ip'])
# GitHub publish event
if ('action' in event['body-json'] and event['body-json']['action'] == 'published'):
branch_name = 'tags/%s' % event['body-json']['release']['tag_name']
repo_name = full_name + '/release'
else:
repo_name = full_name
try:
# branch names should contain [name] only, tag names - "tags/[name]"
branch_name = event['body-json']['ref'].replace('refs/heads/', '').replace('refs/tags/', 'tags/')
except KeyError:
try:
# Bibucket server
branch_name = event['body-json']['push']['changes'][0]['new']['name']
except:
# Bitbucket Server v6.6.1
try:
branch_name = event['body-json']['changes'][0]['ref']['displayId']
except:
branch_name = 'master'
try:
# GitLab
remote_url = event['body-json']['project']['git_ssh_url']
deploymentname = event['body-json']['repository']['name']
if deploymentname == 'gateway-api':
deploymentname = 'ftl-gateway'
except Exception:
try:
remote_url = 'git@' + event['body-json']['repository']['links']['html']['href'].replace('https://',
'').replace('/',
':',
1) + '.git'
except:
try:
# GitHub
remote_url = event['body-json']['repository']['ssh_url']
except:
# Bitbucket
try:
for i, url in enumerate(event['body-json']['repository']['links']['clone']):
if url['name'] == 'ssh':
ssh_index = i
remote_url = event['body-json']['repository']['links']['clone'][ssh_index]['href']
except:
# BitBucket pull-request
for i, url in enumerate(
event['body-json']['pullRequest']['fromRef']['repository']['links']['clone']):
if url['name'] == 'ssh':
ssh_index = i
remote_url = \
event['body-json']['pullRequest']['fromRef']['repository']['links']['clone'][ssh_index]['href']
try:
codebuild_client = client(service_name='codebuild')
new_build = codebuild_client.start_build(projectName=os.getenv('GitPullCodeBuild'),
environmentVariablesOverride=[
{
'name': 'GitUrl',
'value': remote_url,
'type': 'PLAINTEXT'
},
{
'name': 'Branch',
'value': branch_name,
'type': 'PLAINTEXT'
},
{
'name': 'KeyBucket',
'value': keybucket,
'type': 'PLAINTEXT'
},
{
'name': 'KeyObject',
'value': key,
'type': 'PLAINTEXT'
},
{
'name': 'outputbucket',
'value': outputbucket,
'type': 'PLAINTEXT'
},
{
'name': 'outputbucketkey',
'value': '%s' % (repo_name.replace('/', '_')) + '.zip',
'type': 'PLAINTEXT'
},
{
'name': 'outputbucketpath',
'value': '%s/%s/' % (repo_name, branch_name),
'type': 'PLAINTEXT'
},
{
'name': 'exclude_git',
'value': '%s' % (exclude_git),
'type': 'PLAINTEXT'
}
])
buildId = new_build['build']['id']
logger.info('CodeBuild Build Id is %s' % (buildId))
buildStatus = 'NOT_KNOWN'
counter = 0
while (counter < 60 and buildStatus != 'SUCCEEDED'): # capped this, so it just fails if it takes too long
logger.info("Waiting for Codebuild to complete")
time.sleep(5)
logger.info(counter)
counter = counter + 1
theBuild = codebuild_client.batch_get_builds(ids=[buildId])
print(theBuild)
buildStatus = theBuild['builds'][0]['buildStatus']
logger.info('CodeBuild Build Status is %s' % (buildStatus))
if buildStatus == 'SUCCEEDED':
EnvVariables = theBuild['builds'][0]['exportedEnvironmentVariables']
commit_id = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_ID'][0]['value']
commit_message = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_MSG'][0]['value']
current_revision = {
'revision': "Git Commit Id:" + commit_id,
'changeIdentifier': 'GitLab',
'revisionSummary': "Git Commit Message:" + commit_message
}
outputVariables = {
'commit_id': "Git Commit Id:" + commit_id,
'commit_message': "Git Commit Message:" + commit_message
}
break
elif buildStatus == 'FAILED' or buildStatus == 'FAULT' or buildStatus == 'STOPPED' or buildStatus == 'TIMED_OUT':
break
except Exception as e:
logger.info("Error in Function: %s" % (e))