Designing a build and deploy pipeline you'll love for years

I've been working with Azure DevOps pipelines in YAML for quite a while, experimenting with different approaches. Some turned out great, others not so much, but each one taught me valuable lessons. In this post, I'll share how I design pipelines that I love and that are easy to maintain.

In this post I will give examples which is specific for Azure DevOps pipelines, which is written in YAML. But the principles can be applied to Github Actions, Jenkins, or any other CI/CD tool that supports pipelines as code.

Designing a pipeline

A project can have multiple pipelines. Each pipeline can have different purposes, but they also could have some shared common steps. For example, you might want to build and test your code in every pipeline and deploy it to staging and production in the CICD pipeline.

You don't want to repeat yourself (DRY) by writing the same steps in every pipeline. Instead, you can use templates to define common steps. Templates can be used to define a set of steps that can be reused in multiple pipelines.

I like to split up specific parts of the pipelines into stages. For example, I have a stage for building and unittesting the code, a stage for deployment and a stage for quality checks. This way I can easily see what the pipeline is doing and I can easily add or remove stages.

Directory structure

A big part of designing a pipeline that you still love in a year, is on how easy it is to find specific parts of the pipeline in your repository. Below is an example of a directory structure that I like to use for my pipelines:

.
├── pipelines
│   ├── stages
│   │   ├── stage-build.yml
│   │   ├── stage-quality.yml
│   │   ├── stage-deploy.yml
│   ├── templates
│   │   ├── template-backend-build.yml
|   |   ├── template-backend-quality.yml
|   |   ├── template-backend-deploy.yml
│   │   ├── template-frontend-build.yml
|   |   ├── template-frontend-quality.yml
|   |   ├── template-frontend-deploy.yml
│   ├── pipeline-pullrequest.yml
│   ├── pipeline-cicd.yml

In the following sections, I will explain the templates, stages, and pipelines in more detail.

Templates

In the templates directory you can define the templates that you want to use in your stages. The templates consists of a set of steps that can be reused in multiple stages. Here's an example of a template file:

# File: pipelines/templates/template-backend-build.yml

steps:
  - script: echo "Building the backend"
  - script: echo "Running unittests"

The name of the template file is important because that is setting the boundaries of the responsibility of that template. In this case, the template is responsible for building the backend.

You can also argue that the template filename should first start with the action and not the subject (template-build-backend.yml instead of template-backend-build.yml). I think it's a matter of preference.

The template file can also have parameters so you can differ the behavior of the template. Here's an example of a template file with parameters:

# File: pipelines/templates/template-backend-deploy.yml

parameters:
  - name: environment
    type: string
    values:
      - staging
      - production

steps:
  - script: echo "Deploying the backend to ${{ parameters.environment }}"

Stages

In the stages directory you can define the stages of your pipeline. I want to make sure that the stages are reusable. For example, the stage-deploy.yml should be able to deploy to staging and production based on the input parameters.

Here's an example of a stage file:

# File: pipelines/stages/stage-deploy.yml

parameters:
  - name: environment
    type: string
    values:
      - staging
      - production

  - name: stageName
    type: string

  - name: dependsOn
    type: object
    default: []

stages:
  - stage: ${{ parameters.stageName }}
    displayName: "🚀 Deploy to ${{ parameters.environment }}"
    dependsOn:
      - ${{ each dependency in parameters.dependsOn }}:
          - ${{ dependency }}
    jobs:
      - job: DeployBackend
        steps:
          - template: ../templates/template-backend-deploy.yml
            parameters:
              environment: $(environment)
      - job: DeployFrontend
        steps:
          - template: ../templates/template-frontend-deploy.yml
            parameters:
              environment: $(environment)

The parameters section is used to define the input parameters of the stage:

• The environment parameter is used to define the environment where the code is deployed to.
• The stage parameter is used to define the stage name itself, so a single pipeline can use multiple times the same stage (for example, deploy to staging and production).
• The dependsOn parameter is used to define which stages this stage depends on (the build stage in this case).

Pipelines

The last part is the pipeline files. The pipelines ideally should only define the stages and the input parameters of the stages. Here's an example of a pipeline file:

# File: pipelines/pipeline-cicd.yml

stages:
  - template: stages/stage-build.yml

  - template: stages/stage-quality.yml
    parameters:
      stageName: Quality
      dependsOn:
        - Build

  - template: stages/stage-deploy.yml
    parameters:
      environment: staging
      stageName: DeployStaging
      dependsOn:
        - Build
        - Quality

  - template: stages/stage-deploy.yml
    parameters:
      environment: production
      stageName: DeployProduction
      dependsOn:
        - Build
        - Quality
        - DeployStaging

Other tips

• Don't do too much PowerShell or Bash scripting in your pipeline because (a) it's hard to test and (b) it's hard to maintain. Instead, maybe you can create a small console application which you can run in your pipeline.
• You can use the Azure Pipeline extension for Visual Studio Code to get syntax highlighting and autocompletion for your pipeline files.
• Use emoijs in the stage names to make it easier to see what the pipeline is doing at a glance (🏭 Build, 🚀 Deploy, 🧪 Quality, etc.)
• The biggest challenge I'm facing with pipelines is testing. Pushing code changes, wait for the pipeline to run, and then see if it works. This can take a lot of time. Some way to solve this problem is to comment out the steps that you are not working on and then run the pipeline. This way you can test the pipeline faster.

Conclusion

Designing a build and deploy pipelines that you'll love for years is a matter of making it easy to maintain and easy to understand. By splitting up the pipeline in stages and templates, you can make sure that the pipeline is easy to understand and easy to maintain.