Workflows

Goal: High-level understanding of what capabilities workflows provide, how to choose the right workflow engine for your work, where you can find and run existing workflows, and what is involved in using them.

What are workflows and what capabilities do they provide?

A computational workflow is a sequence of computational steps that are used to process data. It is a formalized description of how data is input, how it flows between steps, and how it is output. Computational workflows are widely used in data analysis, scientific computing, and engineering.

Using workflows for your analyses offers the following advantages over running computational steps individually:

• Reproducibility: Using workflows helps ensure that the results of an analysis can be reproduced by others. This is important for scientific research, where it is essential that the results of an experiment can be verified by other researchers.
• Portability and sharing: If you use a widely-supported workflow language, you can run the same workflow on different platforms, which gives you the freedom to choose the platform you want to use. It also enables you to share your workflows with others who use different platforms, and to use workflows developed by others in the community.
• Scalability: Workflows enable you to handle large datasets by automating execution and reducing the possibility of error.

Verily Workbench provides built-in support for running and monitoring WDL-based workflows via the Cromwell workflow engine. Right within the UI, you can add workflows, run them with a set of inputs, and monitor their execution. Visit Using the Cromwell engine to run WDL workflows on Workbench for details.

In addition, you can run dsub and Nextflow workflows via Workbench. See Workflows in Verily Workbench for a discussion of their capabilities and how to choose a workflow engine.

Detailed documentation