Waypoint Custom Pipelines

Before the introduction of Pipelines, Waypoint had a relatively fixed built-in pipeline with the waypoint up CLI command. An "Up" is a special operation that encapsulates the Build, Deploy, and Release workflow for an application. This simple yet powerful pipeline is great for most early users of Waypoint to continuously build and deploy an application to production.

As our product grew, so did our users. Users wished for more fine-tuned control and customization over how Waypoint deploys applications. Teams need to run tests after a build, security scan artifacts prior to a deploy, and so on. Custom Pipelines enable teams to solve increasingly complex challenges. In the Tech Preview, users can define pipelines to run in the order they require to deliver their application. Additionally, users can run scripts as a pipeline execution step via the "Exec" plugin, which further enables highly customized operations.

Pipelines allow users to tell Waypoint what order these operations should run in and configure any operation options necessary; such as disabling an artifact push in the build step, or pruning old deployments on a release in the release step.

Please visit the Pipeline Configuration Docs for more information about the configuration of a Pipeline for a Waypoint project.

Pipeline Runs

Each time a pipeline is run, Waypoint keeps track of the steps, jobs, and run results. As you can see from the waypoint pipeline run console example output, Waypoint identifies how many steps are in the pipeline and how many are successfully completed. Users can then further deep dive into specific pipeline runs to view more information.

waypoint pipeline list shows the current steps, last run start and complete times, last run state, and total runs for every pipeline in the project.

❯ waypoint pipeline list

» Waypoint Pipelines for my-project
              ID             | NAME |    OWNER     | CURRENT STEPS | LAST RUN STARTED | LAST RUN COMPLETED |  STATE  | TOTAL RUNS
-----------------------------+------+--------------+---------------+------------------+--------------------+---------+-------------
  01GBBBC63TR5DYQXQVGJPS8JGZ | test | example-java |             2 | 5 days ago       | 5 days ago         | success |          5

waypoint pipeline inspect can look up a specific pipeline by ID (default) or name, as well as an additional option to view a specific run of that pipeline. Pipeline run information will include run state, steps, timestamps, and associated job IDs.

❯ waypoint pipeline inspect -name=test


» Pipeline Configuration
                  ID: 01GBBBC63TR5DYQXQVGJPS8JGZ
                Name: test
               Owner: example-java
      Root Step Name: build
         Total Steps: 2
          Total Runs: 9
    Last Run Started: 1 minute ago
  Last Run Completed: a long while ago
     Last Run Status: RUNNING

❯ waypoint pipeline inspect -name=test -run=3

» Pipeline Configuration
              ID: 01GBBBC63TR5DYQXQVGJPS8JGZ
            Name: test
           Owner: example-java
  Root Step Name: build
     Total Steps: 2
    Run Sequence: 3
     Jobs Queued: [id:"01GBBBQJZHZVMKRAQSXGYXKEZY" id:"01GBBBQJZH05YD6MN91X30G7HF"]
     Run Started: 5 days ago
   Run Completed: 5 days ago
           State: ERROR

All endpoints and CLI commands related to the running of pipelines: tasks, jobs, deployments, releases, and builds will reflect pipeline and pipeline run information. See waypoint artifact list-builds below for an example:

❯ waypoint artifact list-builds

» example-java
    | ID | WORKSPACE | BUILDER |   STARTED   |  COMPLETED  |        PIPELINE
----+----+-----------+---------+-------------+-------------+----------------------------
  ✔ | 18 | default   | pack    | 3 hours ago | 3 hours ago | test[run: 2][step: build]
  ✔ | 17 | default   | pack    | 4 hours ago | 4 hours ago | test[run: 1][step: build]

Exec Step Plugin

Pipelines come with a built-in step plugin named exec. This plugin allows a step to execute bash within the Pipeline run. This could be used to run tests after a build or deploy, or use tools packaged into the exec step image to security scan the built artifact.

pipeline "example" {
  step "run-some-code" {
    image_url = "https://localhost:5000/waypoint-odr:dev"

    use "exec" {
      # executes a binary test with some arguments
      command = "test"
      args    = ["--full", "test-all"]
    }
  }
}

Embedded Pipelines in Steps

You can optionally define a full pipeline within a Pipeline Step. This pattern is useful if you have a series of steps that make sense to be grouped together, such as doing a deploy and release within a specific workspace.

pipeline "simple-nested" {
  step "build" {
    use "build" {}
  }

  step "deploy" {
    pipeline "deploy" {
      step "deploy" {
        use "deploy" {
        }
      }
      step "release" {
        use "release" {
          prune = true
        }
      }
    }
  }
}

Reference Pipelines As Steps

You can reference other Pipelines within a Project via a pipeline step:

project = "my-pipeline-project"

pipeline "simple-nested" {
  step "build" {
    use "build" {}
  }

  step "my-deploy" {
    #  "pipeline" is a special "built-in" step that references another pipeline
    use "pipeline" {
      project = "my-pipeline-project"
      name    = "deploy"
    }
  }
}

pipeline "deploy" {
  step "deploy" {
    use "deploy" {}
  }
  step "release" {
    use "release" {
      prune = true
    }
  }
}

Pipelines Across Workspaces

Pipelines can be defined and scoped to a Workspace for deploying in various environments. When you use the pipeline run CLI, you can run a pipeline in a different workspace with the -workspace flag.

$ waypoint pipeline run -workspace=dev builder

Workspace Scoped Steps

You can override the environment that a Pipeline step runs in by specifying a workspace param for any step in your Pipeline definition. For example, this pipeline's "build-prod" step will run in the production ("prod") workspace.

Notice that the "build" stanza in the "app" definition contains two "use" plugin definitions, one of which is workspace scoped. That means when the "build-prod" step runs, it will use the "prod" workspace scoped build plugin pack rather than the default "docker" plugin.

// waypoint.hcl
pipeline "builder" {
  step "build-dev" {
    use "build" {}
  }
  step "build-prod" {
    workspace = "prod"
    use "build" {}
  }
  step "deploy-dev" {
    use "deploy" {
      release = false
    }
  }
}

// waypoint.hcl
app "tetris" {
  build {
    use "docker" {}

    workspace "prod" {
      use "pack" {}
    }
  }
  deploy {
    use "docker" {}

    workspace "prod" {
      use "kubernetes" {}
    }
  }
}

You can also use this to set a workspace for an embedded pipeline:

pipeline "builder" {
  step "build" {
    use "build" {}
  }
  step "deploy-test" {
    workspace = "test"
    pipeline "deploy-test-env" {
      step "build" {
        // test env uses docker-pull
        use "build" {}
      }
      step "deploy" {
        use "deploy" {}
      }
      // more steps here...
    }
  }
  step "deploy-prod" {
    workspace = "prod"
    pipeline "deploy-prod-env" {
      step "build" {
        // prod env uses docker-pull
        use "build" {}
      }
      step "deploy" {
        use "deploy" {}
      }
      // more steps here...
    }
  }
}

Examples

Below is an example that redefines the Up operation as a pipeline with three steps for a build, deploy, and release workflow. This example could be used as a starting point for more complicated pipelines.

project = "my-project"

pipeline "full-up" {
  step "my-build" {
    use "build" {}
  }

  step "my-deploy" {
    use "deploy" {}
  }

  step "my-release" {
    use "release" {}
  }
}

app "my-app" {
  build { ... }
  deploy { ... }
  release { ... }
}

The exec step accepts "command" and "arg" parameters for executing commands.

pipeline "build-and-test" {
  step "my-build" {
    use "build" {}
  }
  step "test" {
    image_url = "https://localhost:5000/waypoint-odr:dev"
    use "exec" {
      # executes a binary test with some arguments
      command = "test"
      args    = ["--full", "test-all"]
    }
  }
  step "my-deploy" {
    use "deploy" {}
  }
}

To run the example pipeline named "full-up" defined in this example, it is as simple as waypoint pipeline run full-up:

❯ waypoint pipeline run full-up

» Running pipeline "full-up" for application "tetris"
✓ Pipeline "full-up" has started running. Attempting to read job stream sequentially in order
✓ 3 steps detected, run sequence 2

  Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"

» Cloning data from Git
  URL: https://github.com/briancain/waypoint-tetris.git

» Downloading from Git
  Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
   Timestamp: 2022-08-29 17:04:04 +0000 UTC
     Message: Adding example pipeline


» Executing Step "build"
  Reading job stream (jobId: 01GBNAQRW2M6TTZ4C0JKT0AXRD)...

  Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"

» Cloning data from Git
  URL: https://github.com/briancain/waypoint-tetris.git

» Downloading from Git
  Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
   Timestamp: 2022-08-29 17:04:04 +0000 UTC
     Message: Adding example pipeline

✓ Running build v1
✓ Building Docker image with kaniko...
✓ Testing registry and uploading entrypoint layer
✓ Executing kaniko...
 │ INFO[0001] Retrieving image manifest nginx:stable
 │ INFO[0001] Returning cached image manifest
 │ INFO[0001] Executing 0 build triggers
 │ INFO[0001] Unpacking rootfs as cmd COPY ./public/ /var/www requires it.
 │ INFO[0004] COPY ./public/ /var/www
 │ INFO[0004] Taking snapshot of files...
 │ INFO[0004] COPY ./nginx.conf /etc/nginx/conf.d/default.conf
 │ INFO[0004] Taking snapshot of files...
 │ INFO[0004] Pushing image to localhost:32953/tetris:latest
 │ INFO[0005] Pushed image to 1 destinations
✓ Image pushed to 'my-container-registry/tetris:latest'
✓ Running push build v1

» Executing Step "deploy"
  Reading job stream (jobId: 01GBNAQRW29Q1M7QNQ0F1JX2CS)...

  Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"

» Cloning data from Git
  URL: https://github.com/briancain/waypoint-tetris.git

» Downloading from Git
  Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
   Timestamp: 2022-08-29 17:04:04 +0000 UTC
     Message: Adding example pipeline

✓ Running deploy v1
✓ Kubernetes client connected to https://10.96.0.1:443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "tetris-v1"
✓ Deployment successfully rolled out!
✓ Horizontal Pod Autoscaler has been created

» Executing Step "release"
  Reading job stream (jobId: 01GBNAQRW22N7PQXSF2KGXT1MS)...

  Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"

» Cloning data from Git
  URL: https://github.com/briancain/waypoint-tetris.git

» Downloading from Git
  Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
   Timestamp: 2022-08-29 17:04:04 +0000 UTC
     Message: Adding example pipeline

✓ Running release v1
✓ Kubernetes client connected to https://my-kubernetes.example.com:443 with namespace default
✓ Creating service...
✓ Service is ready!
✔ Pipeline "full-up" (k8s-tetris) finished! 3/3 steps successfully completed.

Future Roadmap For Pipelines

We plan to continue making improvements to pipelines in the coming months to bring it out of Tech Preview. This includes more support for running batches of operations for a pipeline by improving Waypoint's internal job system, sharing VCS project source code between Steps, giving Pipeline Steps the ability to share input and output variables, approval Steps to "manually" sign off on releases, and more!

Current open feature requests, bug reports, and general issues can be found via the core/pipeline label on GitHub!