Transformer course business dialogue robot rasa 3 X continuous integration and continuous deployment
Continuous Integration and Continuous Deployment
Even if the development context assistant is different from developing traditional software, you should still follow software development best practices. Setting up continuous integration and continuous deployment pipelines ensures that incremental updates to the bot can improve it without compromising it.
Overview
Continuous integration is the practice of merging code changes frequently and automatically testing changes when they are submitted. Continuous deployment means that integrated changes are automatically deployed to staging or production environments. They enable you to improve assistants more frequently and effectively test and deploy these changes.
This guide will cover what should be included in the continuous integration and continuous deployment pipeline specific to the Rasa project. It is recommended that you choose a tool that integrates with any Git repository you use.
Continuous Integration
The best way to improve the assistant is to make frequent incremental updates. No matter how small the change is, make sure it doesn't create new problems or negatively affect the assistant's performance.
It is usually best to run checks when merging / pulling requests or submitting. Most tests are fast enough to run with each change. However, you can choose to run resource intensive tests only if some files have changed or other indicators exist. For example, if your code is hosted on Github, you can run the test only if the pull request has a specific tag (such as "NLU test required").
Continuous Integration Pipeline Overview
The continuous integration pipeline should include model training and testing as steps to simplify the deployment process. The first step after saving the new training data is to start the pipeline. This can be started manually or when a request is created or updated.
Next, you need to run various test sets to see the impact of the changes. This includes running tests for data validation, NLU cross validation, and story testing. For more information about testing, see test assistant.
The final step is to check the test results and, if the test is successful, push the change. Once the new model has been trained and tested, it can be deployed automatically using the continuous deployment pipeline.
GitHub Actions CI Pipeline
You can automate data validation, training, and testing using the Rasa Train Test Github operation in the continuous integration pipeline.
An example of a continuous integration pipeline using Github operations is as follows:
jobs:
training-testing:
name: Training and Testing
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Rasa Train and Test GitHub Action
uses: RasaHQ/rasa-train-test-gha@main
with:
requirements_file: requirements.txt
data_validate: true
rasa_train: true
cross_validation: true
rasa_test: true
test_type: all
publish_summary: true
github_token: ${{ secrets.GITHUB_TOKEN }}
- name: Upload model
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@master
with:
name: model
path: models
In this pipeline, the Rasa Train Test Github operation performs data verification, model training and story testing in the first step, and uploads the model file in the second step.
A complete list of configurable parameters for the Rasa Train Test Github operation is provided in the readme file of the repository.
When publish_ When summary is set to true, this operation will automatically publish the test results of the model as comments to the associated Pull request:
Pull requests can be approved or rejected based on the evaluation results, and in many cases, if all checks pass, you will want to automate the deployment of the model. You can continue to the next section to learn more about continuous deployment.
Continuous Deployment
To provide improvements to users on a regular basis, you need to automate the deployment process as much as possible.
Once the check is successful, the continuous deployment step is usually run when pushing or merging to a branch.
Deploying Your Rasa Model
If you run a test story in the continuous integration pipeline, you already have a trained model. If the continuous integration results are satisfactory, you can set the continuous deployment pipeline and upload the trained model to the Rasa server. For example, to upload a model to Rasa X:
curl -k -F "model=@models/my_model.tar.gz" "https://example.rasa.com/api/projects/default/models?api_token={your_api_token}"
If you are using Rasa X, you can also mark the uploaded model as production (or any deployment to mark when using multiple deployment environments):
curl -X PUT "https://example.rasa.com/api/projects/default/models/my_model/tags/production"
Update of the code of conduct
If the update includes changes to the model and operation code, and these changes are interdependent in any way, the model should not be automatically marked as production. You first need to build and deploy the updated operation server so that the new model will not invoke operations that did not exist in the operation server before the update.
Deploying Your Action Server
For each update of the operation code, you can automatically build a new image for the operation server and upload it to the image repository. As mentioned above, if the operation server is not compatible with the current production model, be careful to automatically deploy new image tags in production.
As an example, see the Rara Rasa assistant and the Carbon Bot. Both use Github as a continuous integration and continuous deployment tool.
These examples are just two of many possibilities. If you have your favorite continuous integration and continuous deployment settings, please share them with the Rasa community on the forum.
https://github.com/RasaHQ/carbon-bot/blob/master/.github/workflows/model_ci.yml
name: Model CI
on:
push:
branches:
- 'master'
pull_request:
types: [opened, synchronize, reopened]
env:
DOMAIN: 'carbon.rasa.com'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_NAME: carbon-assistant
NAMESPACE: carbon-assistant
RASA_X_IMAGE_NAME: ${{ secrets.RASA_X_IMAGE_NAME }}
RASA_X_USERNAME: admin
RASA_X_PASSWORD: ${{ secrets.RASA_X_ADMIN_PASSWORD }}
jobs:
build-model:
name: Build, test, and upload model
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements.txt
- name: Check stories are consistent
run: |
rasa data validate stories --max-history 5 --fail-on-warning
- name: Train model
run: |
rasa train
- name: Run Through Test Stories
run: |
rasa test core --stories test_stories/stories.yml --fail-on-prediction-errors
- name: Cross-validate NLU model
id: cvnlu
if: github.event_name == 'pull_request'
run: |
rasa test nlu -f 2 --cross-validation
python format_results.py
- name: Upload Cross Validation Results
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v2
with:
name: cross-validation-result
path: results.md
- name: Upload model
if: github.event_name == 'push' && (startsWith(github.event.ref, 'refs/tags') || github.ref == 'refs/heads/master')
uses: actions/upload-artifact@v2
with:
name: model
path: models
- name: Post cross-val results to PR
if: steps.cvnlu.outcome == 'success'
uses: samsucik/comment-on-pr@comment-file-contents
continue-on-error: true
with:
msg: results.md
build-images:
name: Build and Push Images
runs-on: ubuntu-latest
env:
IS_PUSH_EVENT: ${{ github.event_name == 'push' }}
steps:
- uses: actions/checkout@v2
- name: Set image tag
run: |
if [[ $IS_PUSH_EVENT == "false" ]]
then
IMAGE_TAG=${{ github.head_ref }}
else
IMAGE_TAG=$(basename ${{ github.ref }})
fi
echo "IMAGE_TAG=${IMAGE_TAG}-${{ github.sha }}" >> $GITHUB_ENV
# Setup gcloud CLI
- uses: google-github-actions/setup-gcloud@v0.2.1
name: Google Auth
with:
service_account_key: ${{ secrets.GKE_SA_KEY }}
project_id: ${{ secrets.GKE_PROJECT }}
# Configure docker to use the gcloud command-line tool as a credential helper
- run: |-
gcloud --quiet auth configure-docker
- name: Build and push the Docker image
run: |
# Read and export variables from .env file
set -o allexport; source .env; set +o allexport
docker build . \
--build-arg RASA_SDK_IMAGE=rasa/rasa-sdk:${RASA_SDK_VERSION} \
--tag gcr.io/${{ secrets.GKE_PROJECT }}/carbon-bot-actions:$IMAGE_TAG
docker push gcr.io/${{ secrets.GKE_PROJECT }}/carbon-bot-actions:$IMAGE_TAG
deploy-to-cluster:
name: Re-deploy the cluster with the latest action server
runs-on: ubuntu-latest
needs:
- build-images
if: github.event_name == 'push' && (startsWith(github.event.ref, 'refs/tags') || github.ref == 'refs/heads/master')
steps:
# Checkout repository because we need the content of the `.env` file later
- uses: actions/checkout@v2
- name: Set image tag
env:
IS_PUSH_EVENT: ${{ github.event_name == 'push' }}
run: |
if [[ $IS_PUSH_EVENT == "false" ]]
then
IMAGE_TAG=${{ github.head_ref }}
else
IMAGE_TAG=$(basename ${{ github.ref }})
fi
echo "IMAGE_TAG=${IMAGE_TAG}-${{ github.sha }}" >> $GITHUB_ENV
# Setup gcloud CLI
- uses: google-github-actions/setup-gcloud@v0.2.1
name: Google Auth
with:
service_account_key: ${{ secrets.GKE_SA_KEY }}
project_id: ${{ secrets.GKE_PROJECT }}
# Configure docker to use the gcloud command-line tool as a credential helper
- run: |-
gcloud --quiet auth configure-docker
# Get the GKE credentials so we can deploy to the cluster
- uses: google-github-actions/get-gke-credentials@v0.2.1
with:
cluster_name: ${{ secrets.GKE_CLUSTER }}
location: ${{ secrets.GKE_ZONE }}
credentials: ${{ secrets.GKE_SA_KEY }}
- name: Install Helm and helmfile ⛑
run: |
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
sudo curl -fsSL https://github.com/roboll/helmfile/releases/download/v0.138.7/helmfile_linux_amd64 --output /usr/local/bin/helmfile
sudo chmod +x /usr/local/bin/helmfile
- name: Install Chart
env:
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
RASA_TOKEN: ${{ secrets.RASA_TOKEN }}
RASA_X_TOKEN: ${{ secrets.RASA_X_TOKEN }}
JWT_SECRET: ${{ secrets.JWT_SECRET }}
PASSWORD_SALT: ${{ secrets.PASSWORD_SALT }}
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD }}
RABBITMQ_PASSWORD: ${{ secrets.RABBITMQ_PASSWORD }}
RASA_X_ADMIN_PASSWORD: ${{ secrets.RASA_X_ADMIN_PASSWORD }}
GKE_PROJECT: ${{ secrets.GKE_PROJECT }}
FACEBOOK_VERIFY_TOKEN: ${{ secrets.FACEBOOK_VERIFY_TOKEN }}
FACEBOOK_APP_SECRET: ${{ secrets.FACEBOOK_APP_SECRET }}
FACEBOOK_PAGE_ACCESS_TOKEN: ${{ secrets.FACEBOOK_PAGE_ACCESS_TOKEN }}
CLIMATIQ_API_KEY: ${{ secrets.CLIMATIQ_API_KEY }}
run: |
# Install helm v3
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
# Read and export variables from .env file
set -o allexport; source .env; set +o allexport
kubectl create ns ${NAMESPACE} || true
cd .github/deployments; helmfile sync
cat <<EOF | kubectl apply --namespace ${NAMESPACE} -f -
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
generation: 1
name: rasa-bots-certificate
spec:
domains:
- ${{ env.DOMAIN }}
EOF
upload-model:
name: Upload the trained model to Rasa X
needs:
- deploy-to-cluster
- build-model
env:
MODEL_DIRECTORY: "models"
if: github.event_name == 'push' && (startsWith(github.event.ref, 'refs/tags') || github.ref == 'refs/heads/master')
runs-on: ubuntu-latest
steps:
- name: Download Model
uses: actions/download-artifact@v2
with:
name: model
path: ${{ env.MODEL_DIRECTORY }}
- name: Get path to model
run: |
ls -R
echo "MODELNAME=${{ env.MODEL_DIRECTORY }}/$(ls ${{ env.MODEL_DIRECTORY }})" >> $GITHUB_ENV
- name: Upload Model to Rasa
run: |
# Get token
RASA_X_TOKEN=$(curl -s --header "Content-Type: application/json" \
--request POST \
--data "{\"username\":\"${RASA_X_USERNAME}\",\"password\":\"${RASA_X_PASSWORD}\"}" \
https://${{ env.DOMAIN }}/api/auth | jq -r .access_token)
# Upload model
curl -k --fail -H "Authorization: Bearer ${RASA_X_TOKEN}" -F "model=@${MODELNAME}" https://${{ env.DOMAIN }}/api/projects/default/models
# ensure model is ready and tag as production
sleep 5
export MODEL=$(basename ${MODELNAME} .tar.gz)
curl --fail -XPUT -H "Authorization: Bearer ${RASA_X_TOKEN}" https://${{ env.DOMAIN }}/api/projects/default/models/${MODEL}/tags/production
https://github.com/RasaHQ/rasa-demo/blob/main/.github/workflows/continuous_integration.yml
name: Continuous Integration
on: [pull_request]
env:
GDRIVE_CREDENTIALS: ${{ secrets.GDRIVE_CREDENTIALS }}
MAILCHIMP_LIST: ${{ secrets.MAILCHIMP_LIST }}
MAILCHIMP_API_KEY: ${{ secrets.MAILCHIMP_API_KEY }}
ALGOLIA_APP_ID: ${{ secrets.ALGOLIA_APP_ID }}
ALGOLIA_SEARCH_KEY: ${{ secrets.ALGOLIA_SEARCH_KEY }}
ALGOLIA_DOCS_INDEX: ${{ secrets.ALGOLIA_DOCS_INDEX }}
RASA_X_HOST: ${{ secrets.RASA_X_DOMAIN }}
RASA_X_PASSWORD: ${{ secrets.RASA_X_PASSWORD }}
RASA_X_USERNAME: ${{ secrets.RASA_X_USERNAME }}
RASA_X_HOST_SCHEMA: ${{ secrets.RASA_X_HOST_SCHEMA }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TRACKER_DB_URL: ${{ secrets.TRACKER_DB_URL }}
# Due to the issue with openssl library for Google Cloud SDK (gcloud)
# (https://github.com/GoogleCloudPlatform/github-actions/issues/128)
# we use 297.0.01 version
GCLOUD_VERSION: "297.0.1"
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
check_changed_files:
name: Check for file changes
runs-on: ubuntu-20.04
outputs:
nlu: ${{ steps.changed-files.outputs.nlu }}
core: ${{ steps.changed-files.outputs.core }}
training: ${{ steps.changed-files.outputs.training }}
actions: ${{ steps.changed-files.outputs.actions }}
steps:
# Due to an issue with checking out a wrong commit, we make sure
# to checkout HEAD commit for a pull request.
# More details: https://github.com/actions/checkout/issues/299
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- uses: RasaHQ/pr-changed-files-filter@c4f7116a04b8a4596313469429e2ad235f59d9c4
id: changed-files
with:
token: ${{ secrets.GITHUB_TOKEN }}
filters: .github/change_filters.yml
base: ${{ github.ref }}
lint-testing:
name: Code Formatting Tests
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Code Formatting Tests
run: |
echo "------------------------------------"
echo "/usr/bin/git log -1 --format='%H'"
/usr/bin/git log -1 --format='%H'
echo "------------------------------------"
make lint
type-testing:
name: Type Tests
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Type Checking
run: |
pip list
make types
action-unit-tests:
needs:
- lint-testing
- type-testing
name: Custom Action Unit Tests
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
make install-dev
- name: Unit Tests
run: |
make test-actions
data-validation:
name: Data Validation
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
- name: Rasa Data Validation
run: |
rasa data validate --debug
training-testing:
name: Test Model
runs-on: ubuntu-latest
needs:
- data-validation
- check_changed_files
if: ${{ needs.check_changed_files.outputs.training == 'true' }}
steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.8.0
with:
access_token: ${{ github.token }}
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade "pip<20"
pip install -r requirements-dev.txt
rasa --version
- name: Cross-validate NLU model
id: cvnlu
if: contains( github.event.pull_request.labels.*.name, 'nlu_testing_required' )
run: |
rasa --version
rasa test nlu -f 3 --cross-validation --config config_nlu_testing.yml
python .github/workflows/format_results.py
- name: post cross-val results to PR
if: steps.cvnlu.outcome == 'success'
uses: amn41/comment-on-pr@comment-file-contents
continue-on-error: true
with:
msg: results.md
- name: Train Model
run: |
rasa --version
rasa train
- name: Test End 2 End Stories
if: ${{ needs.check_changed_files.outputs.core == 'true' }}
run: |
rasa --version
rasa test core --stories tests/test_conversations.yml --fail-on-prediction-errors
- name: Wait for the conclusion of all other workflow runs
# upload model from PR
if: github.event_name == 'pull_request'
id: check-runs-conclusion
env:
WAIT_INTERVAL_SECS: 10
timeout-minutes: 20
run: |
while true; do
# Get a list of checks information, excluding training-testing and build-images
CHECKS_LIST=$(gh api /repos/${{ github.repository }}/commits/${{ github.event.pull_request.head.sha }}/check-runs --jq '.check_runs.[] | select(.name != "Test Model" and .name != "Build Action Server Image")')
# Get the status and conclusion of echo jobs
STATUS_LIST=$(echo $CHECKS_LIST | jq -r '.status')
CONCLUSION_LIST=$(echo $CHECKS_LIST | jq -r '.conclusion')
# Make sure all other check runs are completed
if [[ "$(echo $STATUS_LIST | tr ' ' '\n' | sort | uniq)" == "completed" ]]; then
# Check the conclusion of all other check runs
# Fail the step if there is any failture
if [[ "$(echo CONCLUSION_LIST | tr ' ' '\n' | sort | uniq)" =~ "failure" ]]; then
echo "::error:: Some check runs failed. Skip uploading model."
exit 1
else
echo "All other check runs are successed."
echo "::set-output name=upload-model::true"
exit 0
fi
fi
sleep $WAIT_INTERVAL_SECS
echo "Wait for $WAIT_INTERVAL_SECS seconds..."
done
- name: Set model name from Rasa version
if: |
github.event_name == 'pull_request' &&
steps.check-runs-conclusion.outputs.upload-model == 'true'
run: |
python -c "import rasa; open('rasaversion.txt','w+').write(rasa.__version__)"
rasa_version=`cat rasaversion.txt`
model_path=`ls models/*.tar.gz | head -n 1`
model_timestamp=$(basename "$model_path" .tar.gz)
model_name="$model_timestamp"_rasa"$rasa_version"
renamed_model_path=models/"$model_name".tar.gz
mv $model_path $renamed_model_path
echo "MODEL_NAME=${model_name}" >> $GITHUB_ENV
echo "MODEL_PATH=${renamed_model_path}" >> $GITHUB_ENV
- uses: google-github-actions/setup-gcloud@master
if: |
(github.event_name == 'pull_request' &&
contains( github.event.pull_request.labels.*.name, 'upload_model' )) ||
steps.check-runs-conclusion.outputs.upload-model == 'true'
name: Authenticate with gcloud 🎫
with:
version: "${{ env.GCLOUD_VERSION }}"
service_account_email: ${{ secrets.SARA_GKE_SERVICE_ACCOUNT_NAME }}
service_account_key: ${{ secrets.SARA_GKE_SERVICE_ACCOUNT_KEY }}
- name: Upload model to storage bucket
if: |
(github.event_name == 'pull_request' &&
contains( github.event.pull_request.labels.*.name, 'upload_model' )) ||
steps.check-runs-conclusion.outputs.upload-model == 'true'
run: gsutil cp "${MODEL_PATH}" ${{ secrets.STORAGE_BUCKET_URL }}/rasa_demo_models
build-images:
name: Build Action Server Image
needs:
- lint-testing
- type-testing
- training-testing
- check_changed_files
if: ${{ needs.check_changed_files.outputs.actions == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Checkout pull request HEAD commit instead of merge commit
uses: actions/checkout@v2
if: github.event_name == 'pull_request'
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: Checkout git repository
uses: actions/checkout@v2
if: github.event_name != 'pull_request'
- name: Authenticate with gcloud 🎫
uses: google-github-actions/setup-gcloud@daadedc81d5f9d3c06d2c92f49202a3cc2b919ba
with:
version: ${{ env.GCLOUD_VERSION }}
service_account_key: ${{ secrets.GCLOUD_AUTH }}
- name: Configure Docker to use Google Cloud Platform
run: |
gcloud auth configure-docker
- name: Pull Latest Image
run: |
docker pull gcr.io/replicated-test/rasa-demo:latest || true
- name: Build Image
run: |
docker build --cache-from gcr.io/replicated-test/rasa-demo:latest .
https://github.com/features/actions
