Automated DevOps for the Serverless Fitness Shop - Knowing what and why you go to production @ NS1 INS1GHTS 2020

Dive into the future of DevOps with Continuous Verification! Learn how automating checks in CI/CD pipelines enhances performance, security, and cost evaluation, reducing manual tasks and ensuring smoother deployments for more beach time!

June 25, 2020

Talks, Serverless, VMware

Automated DevOps for the Serverless Fitness Shop - Knowing what and why you go to production @ NS1 INS1GHTS 2020

Page content

In a nutshell, Continuous Verification comes down to making sure that DevOps teams put as many checks as possible into their CI/CD pipelines. These checks use external systems to validate the performance, security, and cost of your app without asking your engineers to do that manually. The systems that provide the data which decided whether your deployment goes to production or not, can also be used to help your engineers understand where the bottlenecks are in the process. With more checks in your automated pipeline, you have fewer manual tasks, less overhead, and better decisions to deploy to production or not. All that together means you get to spend more time at the beach!

The talk

At VMware we define Continuous Verification as:

“A process of querying external systems and using information from the response to make decisions to improve the development and deployment process.”

Continuous Verification, by default, means it’s an extension to the existing development and deployment processes that companies already have. That extension focuses on optimizing both the development and deployment experience for those companies by looking at security, performance, and cost. At most companies, some of these steps are done manually or scripted, but hardly ever are they really part of the deployment pipeline.

Slides

Talk materials

DevOps Pipeline

## Set the default image for the CI workflow
image: docker:19.03.8

## Global variables available to the workflow
variables:
    ## The host for the docker registry, set to docker:2375 to work with DinD
    DOCKER_HOST: tcp://docker:2375
    ## Skip verification of TLS certificates for DinD
    DOCKER_TLS_CERTDIR: ""

## Specify which GitLab templates should be included
include:
    template: Container-Scanning.gitlab-ci.yml

## Specify the stages that exist in the template and the order in which they need to run
stages:
    - scan_code
    - build
    - container_scanning
    - governance
    - deploy_staging
    - performance
    - deploy_production

## Stage scan_code performs a vulnerability analysis of the code using Snyk.io
scan_code:
    stage: scan_code
    image: golang:1.14
    script:
        ## Download the latest version of the Snyk CLI for Linux
        - curl -o /bin/snyk -L https://github.com/snyk/snyk/releases/latest/download/snyk-linux
        - chmod +x /bin/snyk
        ## Authenticate using a Snyk API token
        - snyk auth $SNYK_TOKEN
        ## Run snyk to test for vulnerabilities in the dependencies
        - snyk test

## Build the container tagged with the commit revision for which project is built
build:
    stage: build
    image: docker:19.03.8
    services:
        - docker:19.03.8-dind
    variables:
        DOCKER_HOST: tcp://docker:2375/
        DOCKER_DRIVER: overlay2
    before_script:
        - docker info
        - docker login -u "$DOCKER_USER" -p "$DOCKER_PASSWORD"
    script:
        - docker info
        - docker build --file /builds/retgits/test/cmd/cloudrun-payment-http/Dockerfile . -t $DOCKER_USER/payment:$CI_BUILD_REF
        - docker push $DOCKER_USER/payment:$CI_BUILD_REF

## Scan containers built in this job
container_scanning:
    stage: container_scanning

## Validate whether the project is still within budget
budget:
    stage: governance
    image: vtimd/alpine-python-kubectl
    script:
        - chmod +x ./governance/budget.py
        - ./governance/budget.py $GITLAB_TOKEN
        - if [ $OVERAGE = "OVER" ]; then exit 1 ; else echo "Within Budget. Continuing!"; fi

## Validate whether the project follows the best practices set by the security team
security:
    stage: governance
    image: vtimd/alpine-python-kubectl
    script:
        - chmod +x ./governance/security.py
        - ./governance/security.py
        - if [ $VSS_VIOLATION_FOUND = "True" ]; then exit 1 ; else echo "Violation Check Passed. Continuing!"; fi

## Deploy the service to staging
deploy_staging:
    stage: deploy_staging
    image: google/cloud-sdk:alpine
    script:
        # Authenticate using the service account
        - echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
        - gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
        - gcloud config set project $GCP_PROJECT_ID
        # Deploy
        - gcloud run deploy payment --namespace=default --image=retgits/payment:6cc4ac945f98f7e2c4770779ff13431e399b9ea6 --platform=gke --cluster=$CLUSTER --cluster-location=$CLUSTER_LOCATION --connectivity=external --set-env-vars=SENTRY_DSN=$SENTRY_DSN,VERSION=$VERSION,STAGE=dev,WAVEFRONT_TOKEN=$WAVEFRONT_TOKEN,WAVEFRONT_URL=$WAVEFRONT_URL,MONGO_USERNAME=$MONGO_USERNAME,MONGO_PASSWORD=$MONGO_PASSWORD,MONGO_HOSTNAME=$MONGO_HOSTNAME

 ## Start traffic generation
traffic:
    stage: performance
    image: alpine:latest
    script:
        ## Download the latest version of Gotling
        - apk add curl
        - curl -o /bin/gotling -L https://github.com/retgits/gotling/releases/download/v0.3-alpha/gotling
        - chmod +x /bin/gotling
        ## Run performance test
        - gotling governance/trafficgen.yaml

## Check performance against Wavefront
perf_stats:
    stage: performance
    image:
        name: retgits/wavefront-pod-inspector:serverless
        entrypoint: [""]
    script:
        - /bin/entrypoint.sh
        - if [ $abc = "failed" ]; then echo "Alert" && exit 1 ; else echo "Within range. Continuing!"; fi

## Deploy the service to production
deploy_production:
    stage: deploy_production
    image: google/cloud-sdk:alpine
    script:
        # Authenticate using the service account
        - echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
        - gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
        - gcloud config set project $GCP_PROJECT_ID
        # Deploy
        - gcloud run deploy payment --namespace=default --image=retgits/payment:6cc4ac945f98f7e2c4770779ff13431e399b9ea6 --platform=gke --cluster=$CLUSTER --cluster-location=$CLUSTER_LOCATION --connectivity=external --set-env-vars=SENTRY_DSN=$SENTRY_DSN,VERSION=$VERSION,STAGE=prod,WAVEFRONT_TOKEN=$WAVEFRONT_TOKEN,WAVEFRONT_URL=$WAVEFRONT_URL,MONGO_USERNAME=$MONGO_USERNAME,MONGO_PASSWORD=$MONGO_PASSWORD,MONGO_HOSTNAME=$MONGO_HOSTNAME