Workflow engines in Workbench: Cromwell, dsub, and Nextflow

Prior reading: Workflows overview

Purpose: This document provides guidance on choosing which workflow engine to use for your deisred purposes.

Introduction

Verily Workbench enables you to run workflows at scale using tools that are popular across the life sciences community. In this document, you’ll discover how to choose from Cromwell, dsub, and Nextflow for your work, and where you can find and run existing workflows.

Choosing a workflow engine

With multiple workflow engines available in Workbench, how do you choose which one to use? Perhaps you want to make use of Workbench's built-in support for Cromwell and Nextflow. In other cases, the decision may be made for you by:

• Standardized engine(s) selected by your organization or team
• Desired tools with existing workflows written for a particular engine

There are a few important things to know about choosing an engine for your work.

Single-stage workflows (tasks)

• Are you only looking to run single-stage "tasks" at scale?
• Are you comfortable with running shell commands, but less comfortable learning new domain-specific languages, such as WDL or Nextflow scripting?

In either of the above cases, dsub may be the right choice for you. dsub is more of a task runner rather than a workflow engine. It doesn’t have built-in capabilities for sequencing multiple workflow steps; instead, where dsub excels is in its simplicity and ease of use.

Multi-stage or multi-environment workflows

If you’re looking to run multi-stage workflows, where each stage runs on a separate VM, and want to take advantage of more sophisticated built-in capabilities (such as automatic horizontal scaling or stage result caching), then Cromwell/WDL and Nextflow may be better choices.

If you are looking to write workflows that you can run in multiple environments, such as Workbench and your institutional HPC cluster, then Cromwell/WDL and Nextflow may again be better choices; both Cromwell and Nextflow support many different backends and executors.

Note that Workbench currently supports Cromwell and Nextflow through the Workbench UI. However, all workflow engines can be executed on VMs within a workspace.

You have flexibility

It’s worth noting that using any one of these workflow engines on Workbench isn’t a deep commitment. These frameworks use the Google Batch API on Google Cloud to execute individual tasks, which means that each is built around:

• Packaged code in Docker images
• Input files localized from Google Cloud Storage
• Output files delocalized to Google Cloud Storage

Moving a workflow (or a subset of workflow tasks) to a different workflow engine can be focused on rewriting the orchestration rather than rewriting individual tasks.

Where to find existing workflows

Cromwell and Nextflow support the orchestration of complex workflows written in the Workflow Definition Language (WDL) and Nextflow language, respectively. WDL and Nextflow are widely used for workflows across the community and many existing workflows can be found on Dockstore:

• WDL workflows
• Nextflow workflows

To learn more about using Cromwell on Workbench, see Use the Cromwell engine to run WDL workflows on Workbench. For a Nextflow tutorial, see Get started with Nextflow on Workbench.