Build AWS codebuild server free cicd interface

Posted by justinjkiss on Tue, 02 Nov 2021 01:59:31 +0100

Requirements: it is used to provide project alternatives with high stability requirements for cicd deployment.
Background: Although Jenkins is superior to codebuild in terms of the process of building cicd by itself, Jenkins will have occasional downtime, resulting in the problem that a large number of services cannot be published. Therefore, it is necessary to have a server free construction method with high stability like codebuild.

Overall architecture

As shown in the figure, a gitlab configured with a corresponding webhook has a push event, which will trigger the gitpull function in lambda to pull the project information of the project. The whole cicd process uses the key to transmit data packets for security. All encryption behaviors are that another function in lambda, createshkey, requests the kms service to create and read the key, Then gitpull sends the pulled git code and all the information of the modified project warehouse (including branches, project names, etc.) to codebuild to build the project. Finally, codebuild pulls the source code of the project, pushes the constructed image to the ecr image warehouse, and operates eks, Publish the latest version of the image in the image warehouse to the corresponding deployment in the cluster.

Create process

AWS officially provides a demo, a cloud formation (AWS service stack) that can synchronize gitlab to s3 bucket and package the source code.
Refer to the official documents for the specific creation process:
During the creation process, pay attention to modifying the availability zone.

After the entire stack is created, modify lambda and codebuild in a customized way. First, the existing condition is that the push event of gitlab library configured with webhook will trigger the operation of lambda. Lambda will send the project information to codebuild, which will use the information to pull the correct branch code for construction. In this analysis, in fact, what needs to be modified most is the content of codebuild. The modification of lambda can be customized according to the needs of later project construction.

The following is the content of codebuild spec

version: 0.2
          python: 3.7
      # commands:
      # - pip3 install boto3
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$      
      - echo "=======================Start-gitpull============================="
      - echo "Getting the SSH Private Key"
      - |
        python3 - << "EOF"
        from boto3 import client
        import os
        s3 = client('s3')
        kms = client('kms')
        enckey = s3.get_object(Bucket=os.getenv('KeyBucket'), Key=os.getenv('KeyObject'))['Body'].read()
        privkey = kms.decrypt(CiphertextBlob=enckey)['Plaintext']
        with open('enc_key.pem', 'w') as f:
            print(privkey.decode("utf-8"), file=f)
      - mv ./enc_key.pem ~/.ssh/id_rsa
      - ls ~/.ssh/
      - echo "Setting SSH config profile"
      - | 
        cat > ~/.ssh/config <<EOF
        Host *
          AddKeysToAgent yes
          StrictHostKeyChecking no
          IdentityFile ~/.ssh/id_rsa
      - chmod 600 ~/.ssh/id_rsa
      - echo "Cloning the repository $GitUrl on branch $Branch"
      - git clone --single-branch --depth=1 --branch $Branch $GitUrl .
      - ls -alh
      - export projectname=$(echo $outputbucketpath | tr '/' ' ' | awk '{print $2}')      
      - |
        if [ "$Branch" = "deploy-staging" ]; then
          export repo_name=$projectname"-staging"
          export env="stage"
        elif [ "$Branch" = "deploy-production" ]; then
          export repo_name=$projectname"-production"
          export env="prod"
          return 1
      - |
        if [ "$projectname" = "export" ]; then
          export contextname="xxxx-bussiness"
        elif [ "$Branch" = "deploy-staging" ]; then
          export contextname="xxxx-bussiness-non-prod"
        elif [ "$Branch" = "deploy-profuction" ]; then
          export contextname="xxx-bussiness"
          return 1
      - export GIT_COMMIT_ID=$(git rev-parse --short HEAD)
      - echo $GIT_COMMIT_ID
      - export GIT_COMMIT_MSG="$(git log -1 --pretty=%B)"
      - echo $GIT_COMMIT_MSG
      - pwd
      - aws s3 cp s3:/xxx/xxx/settings.xml /opt/maven/conf/
      - echo "=======================End-gitgull============================="
      - echo "=======================Start-Build============================="
      - |
        export isjava=$(find ./ -name pom.xml)
        if [ "$isjava" = "" ]; then
          echo "it's node project"
          echo "Start Npm install......................................"
          npm install
          echo "Start Npm run build...................................."
          npm run build:$env
          echo "it's java project"
          mvn  clean package -Dmaven.test.skip=true
      - echo "=======================End-Build============================="
      - echo Build started on `date`
      - echo Building the Docker image...   
      - echo "=====================Start-Docker-build======================"
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$$repo_name:$deploymentname_$GIT_COMMIT_ID
      - echo "=========================End-Docker-build======================"
      - docker image ls
      - mkdir ~/.aws
      - mkdir ~/.kube
      - export imageurl=$AWS_ACCOUNT_ID.dkr.ecr.$$repo_name:$deploymentname_$GIT_COMMIT_ID
      - echo $imageurl
      - aws s3 cp s3://xxx/xxx/kube/ ~/.kube/ --recursive
      - aws s3 cp s3://xxx/xxx/awscli/ ~/.aws/ --recursive
      - aws s3 cp s3://xxx/xxx/ ./
      - aws s3 cp s3://xxx/xxx/image.json ./
      - sed -i 's#imageurl#'$imageurl'#g' image.json
      - sed -i 's#deploymentname#'$deploymentname'#g' image.json
      - ls -alh ~/
      - chmod +x
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$$repo_name:$deploymentname_$GIT_COMMIT_ID
      - kubectl config get-contexts
      - echo $contextname
      - ./ $contextname $deploymentname $repo_name

Problem solving:
1. The permissions of codebuild on ecr, eks and other resources need to be allocated first.
2. How to judge whether the project is a node project or a java project during the construction process.
3. Determine the publishing environment according to different branches.
4. If the maven configuration file involves a custom maven warehouse address, you can copy the configuration file from s3 to the build environment.
5. The configuration files of kubectl and aws are copied locally through s3.
6. The kubectl patch step is documented and scripted to solve the limitation of codebuild spec on cli in this step.

kubectl --context $1 patch deployment $2 --patch "$(cat image.json)" --namespace $3


	"spec": {
		"template": {
			"spec": {
				"containers": [{
					"name": "deploymentname",
					"image": "imageurl"

gitlab permission configuration

Use the PublicSSHKey output by cloudformation to create an ssh key for authentication.

lambda code content

from boto3 import client
import os
import time
import stat
import shutil
from ipaddress import ip_network, ip_address
import logging
import hmac
import hashlib
import distutils.util

exclude_git = bool(distutils.util.strtobool(os.environ['ExcludeGit']))

cleanup = False

key = 'enc_key'

logger = logging.getLogger()
logger.handlers[0].setFormatter(logging.Formatter('[%(asctime)s][%(levelname)s] %(message)s'))

s3 = client('s3')
kms = client('kms')

def lambda_handler(event, context):
    keybucket = event['context']['key-bucket']
    outputbucket = event['context']['output-bucket']
    pubkey = event['context']['public-key']
    # Source IP ranges to allow requests from, if the IP is in one of these the request will not be checked for an api key
    ipranges = []
    if event['context']['allowed-ips']:
        for i in event['context']['allowed-ips'].split(','):
            ipranges.append(ip_network(u'%s' % i))
    # APIKeys, it is recommended to use a different API key for each repo that uses this function
    apikeys = event['context']['api-secrets'].split(',')
    ip = ip_address(event['context']['source-ip'])
    secure = False
    if ipranges:
        for net in ipranges:
            if ip in net:
                secure = True
    if 'X-Git-Token' in event['params']['header'].keys():
        if event['params']['header']['X-Git-Token'] in apikeys:
            secure = True
    if 'X-Gitlab-Token' in event['params']['header'].keys():
        if event['params']['header']['X-Gitlab-Token'] in apikeys:
            secure = True
    if 'X-Hub-Signature' in event['params']['header'].keys():
        for k in apikeys:
            if 'use-sha256' in event['context']:
                k1 ='utf-8'), str(event['context']['raw-body']).encode('utf-8'),
                k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha256=', ''))
                k1 ='utf-8'), str(event['context']['raw-body']).encode('utf-8'),
                k2 = str(event['params']['header']['X-Hub-Signature'].replace('sha1=', ''))
            if k1 == k2:
                secure = True
    # TODO: Add the ability to clone TFS repo using SSH keys
        # GitHub
        full_name = event['body-json']['repository']['full_name']
    except KeyError:
            # BitBucket #14
            full_name = event['body-json']['repository']['fullName']
        except KeyError:
                # GitLab
                full_name = event['body-json']['repository']['path_with_namespace']
            except KeyError:
                    # GitLab 8.5+
                    full_name = event['body-json']['project']['path_with_namespace']
                except KeyError:
                        # BitBucket server
                        full_name = event['body-json']['repository']['name']
                    except KeyError:
                        # BitBucket pull-request
                        full_name = event['body-json']['pullRequest']['fromRef']['repository']['name']
    if not secure:
        logger.error('Source IP %s is not allowed' % event['context']['source-ip'])
        raise Exception('Source IP %s is not allowed' % event['context']['source-ip'])

    # GitHub publish event
    if ('action' in event['body-json'] and event['body-json']['action'] == 'published'):
        branch_name = 'tags/%s' % event['body-json']['release']['tag_name']
        repo_name = full_name + '/release'
        repo_name = full_name
            # branch names should contain [name] only, tag names - "tags/[name]"
            branch_name = event['body-json']['ref'].replace('refs/heads/', '').replace('refs/tags/', 'tags/')
        except KeyError:
                # Bibucket server
                branch_name = event['body-json']['push']['changes'][0]['new']['name']
                # Bitbucket Server v6.6.1
                    branch_name = event['body-json']['changes'][0]['ref']['displayId']
                    branch_name = 'master'
        # GitLab
        remote_url = event['body-json']['project']['git_ssh_url']
        deploymentname = event['body-json']['repository']['name']
        if deploymentname == 'gateway-api':
            deploymentname = 'ftl-gateway'
    except Exception:
            remote_url = 'git@' + event['body-json']['repository']['links']['html']['href'].replace('https://',
                                                                                                                1) + '.git'
                # GitHub
                remote_url = event['body-json']['repository']['ssh_url']
                # Bitbucket
                    for i, url in enumerate(event['body-json']['repository']['links']['clone']):
                        if url['name'] == 'ssh':
                            ssh_index = i
                    remote_url = event['body-json']['repository']['links']['clone'][ssh_index]['href']
                    # BitBucket pull-request
                    for i, url in enumerate(
                        if url['name'] == 'ssh':
                            ssh_index = i

                    remote_url = \
        codebuild_client = client(service_name='codebuild')
        new_build = codebuild_client.start_build(projectName=os.getenv('GitPullCodeBuild'),
                                                         'name': 'GitUrl',
                                                         'value': remote_url,
                                                         'type': 'PLAINTEXT'
                                                         'name': 'Branch',
                                                         'value': branch_name,
                                                         'type': 'PLAINTEXT'
                                                         'name': 'KeyBucket',
                                                         'value': keybucket,
                                                         'type': 'PLAINTEXT'
                                                         'name': 'KeyObject',
                                                         'value': key,
                                                         'type': 'PLAINTEXT'

                                                         'name': 'outputbucket',
                                                         'value': outputbucket,
                                                         'type': 'PLAINTEXT'
                                                         'name': 'outputbucketkey',
                                                         'value': '%s' % (repo_name.replace('/', '_')) + '.zip',
                                                         'type': 'PLAINTEXT'
                                                         'name': 'outputbucketpath',
                                                         'value': '%s/%s/' % (repo_name, branch_name),
                                                         'type': 'PLAINTEXT'
                                                         'name': 'exclude_git',
                                                         'value': '%s' % (exclude_git),
                                                         'type': 'PLAINTEXT'
        buildId = new_build['build']['id']'CodeBuild Build Id is %s' % (buildId))
        buildStatus = 'NOT_KNOWN'
        counter = 0
        while (counter < 60 and buildStatus != 'SUCCEEDED'):  # capped this, so it just fails if it takes too long
  "Waiting for Codebuild to complete")
            counter = counter + 1
            theBuild = codebuild_client.batch_get_builds(ids=[buildId])
            buildStatus = theBuild['builds'][0]['buildStatus']
  'CodeBuild Build Status is %s' % (buildStatus))
            if buildStatus == 'SUCCEEDED':
                EnvVariables = theBuild['builds'][0]['exportedEnvironmentVariables']
                commit_id = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_ID'][0]['value']
                commit_message = [env for env in EnvVariables if env['name'] == 'GIT_COMMIT_MSG'][0]['value']
                current_revision = {
                    'revision': "Git Commit Id:" + commit_id,
                    'changeIdentifier': 'GitLab',
                    'revisionSummary': "Git Commit Message:" + commit_message
                outputVariables = {
                    'commit_id': "Git Commit Id:" + commit_id,
                    'commit_message': "Git Commit Message:" + commit_message
            elif buildStatus == 'FAILED' or buildStatus == 'FAULT' or buildStatus == 'STOPPED' or buildStatus == 'TIMED_OUT':
    except Exception as e:"Error in Function: %s" % (e))

Topics: AWS Lambda