Get started with Nextflow on Verily Workbench

Prior reading: Workflows in Verily Workbench: Cromwell, dsub, and Nextflow

Purpose: This document provides detailed instructions for configuring and running Nextflow pipelines in Verily Workbench.

Introduction

Nextflow is a framework for creating data-driven computational pipelines, as well as scalable and reproducible scientific workflows using software containers.

Workbench supports Nextflow workflows running on Batch in Google Cloud Project-backed workspaces and AWS HealthOmics on Amazon Web Services-backed workspaces. Users can use either the Workbench web UI or Workbench CLI to create and run Nextflow workflows.

Step-by-step guide

Prerequisites

You'll need the following to complete the steps below:

For GCP or AWS workspaces:

• A Nextflow workflow source file (.nf) that either lives in a GCP bucket or S3 folder in your workspace or in a Git repository
• A GCP bucket or S3 folder in your workspace for workflow outputs

For AWS only:

• An external Elastic Container Registry (ECR) repository for your container images since HealthOmics doesn't support public or external container registries. See Configure external ECR repository resources to set one up, and Configure Nextflow for your platform below for other AWS-specific requirements.

In addition, it's highly recommended to have a nextflow.config file within the same folder as the Nextflow workflow source file.

If you intend to use an nf-core pipeline, it's highly recommended to create an Artifact Registry repository. See Configure Nextflow for your platform below for more details.

1. Configure Nextflow for your platform

• GCP
• AWS

us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest

// Set to your registry if you have a mirror of containers
apptainer.registry    = 'us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest'
docker.registry       = 'us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest'
podman.registry       = 'us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest'
singularity.registry  = 'us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest'
charliecloud.registry = 'us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest'

profiles {
  'google-batch' {
    includeConfig "config/google_batch_resources.config"

    process.executor = 'google-batch'
    process.container = "us-central1-docker.pkg.dev/${GOOGLE_CLOUD_PROJECT}/${GOOGLE_ARTIFACT_REPO}/amrplusplus-workbench:latest"
  }
}

2. Create a Nextflow workflow

• Using the UI
• Using the CLI

wb workflow create \
--workflow <YOUR_WORKFLOW_NAME> \
--workflow-type NEXTFLOW \
--bucket-id <NEXTFLOW_BUCKET_NAME> \
--path <NEXTFLOW_BUCKET_PATH>

3. Create a new job

Once you've created a workflow, you can start running jobs.

• Using the UI
• Using the CLI

Select your workflow and click + New job to open the Creating new job dialog. On the Enter job details screen, you can add a Nextflow profile. Profiles are sets of configurations that contain parameters that are used at runtime. Profiles are defined in nextflow.config files within the profiles block. Separate multiple profiles with commas.

Note

Profile order matters differently depending on your platform.

On Batch, profiles are applied in the order they're defined in nextflow.config, regardless of the order you select them here — keep your selection order consistent with your nextflow.config definition order if you rely on profiles overriding each other.

On AWS HealthOmics, profiles are applied in the order you select them here, regardless of nextflow.config definition order (see Configure Nextflow for your platform for details).

The second screen lets you set up parameters and control cache behavior. Enabling the Call Caching toggle enables Nextflow to use the cache and to re-use task outputs. Additionally, you can select a JSON or YAML file with workflow parameters. This file must live in a GCP bucket or S3 folder in your workspace. Click Next.

On the Set up outputs screen, select a GCP bucket or S3 storage folder for your job outputs. A folder name will be auto-generated, but you can update it to a different name.

Note

If you enabled call caching, the chosen output GCP bucket or S3 storage folder will also store your Nextflow cache. To use the cache, ensure that the output resource bucket or folder remains the same between runs of the same workflow.

Click Submit job. You'll be taken to the Job status page where you can select the job you just created and see its details.

Use the wb workflow job run command to start a job.

The example below defines a params file, profile, and the use of Nextflow caching, all of which are optional.

wb workflow job run \
--workflow <WORKFLOW_ID> \ # the ID can be found in the `wb workflow create` output
--job-id <ANY_NAME_FOR_THE_JOB> \
--output-bucket-id <BUCKET_FOR_NEXTFLOW_OUTPUT> \
--output-path <PATH_OF_NEXTFLOW_OUTPUT> \
--params-file-bucket-id <BUCKET_OF_PARAM_FILE> \
--params-file-path <PATH_TO_PARAMS_FILE> \
--profile <PROFILE_FROM_NEXTFLOW.CONFIG> \
--enable-call-caching # whether or not to use Nextflow caching (omit to disable)

4. Monitor your job

Once you've started your job, you can track its progress and view its outputs.

• Using the UI
• Using the CLI

From the Job status tab, click on your job to open its Tasks page, which lists every task in the job.

Each task shows a status chip. You can filter the task list by status:

• Pending – queued, not yet started
• Starting – picked up and initializing
• Running – currently executing
• Completed – finished successfully
• Failed – exited with an error
• Skipped – not run, e.g., due to a when condition
• Cancelled – stopped by a user (via Abort)
• Deleted – removed from the underlying engine

Tasks that are still in progress and can't yet be matched to a specific pipeline step are grouped under an In Progress heading. Once the job finishes, any remaining unmatched tasks are grouped under Ungrouped instead.

If a task's status is Failed, hover over its status chip to see a tooltip suggesting you check the Nextflow logs for more information.

For each task, click View under Logs to see its log output, or View on GCP under Output to browse its output files in the Google Cloud console.

When you run wb workflow job run, you'll see output that includes a job ID, display name, and status.

Run wb workflow job describe with your job ID to get additional information about your job.

Run wb workflow job task list with your job ID to get a list of grouped per process tasks.

Outputs and logs can be viewed on the Workbench web UI. GCP users can also use the Google Cloud console.

Video walkthrough

This video shows how to create a Nextflow workflow using a main.nf workflow source file from a GitHub repository. This repository also includes a nextflow.config file that has a profile called test_full_gcp. We also use a params file in a GCP bucket called nextflow_resources in the workspace.

Troubleshooting

Issues when running large samples

You might see disk space issues and errors if running large samples in GCP. If a task hangs indefinitely and restarting the job does not fix the issue, we recommend making the following modifications to your nextflow.config file:

process {
    cpus   = { 1      * task.attempt }
    memory = { 6.GB   * task.attempt }
    time   = { 4.h    * task.attempt }
    disk   = { 20.GB  * task.attempt }

    errorStrategy = { task.exitStatus in ((130..145) + 104 + (175..177)) ? 'retry' : 'finish' }
    maxRetries    = 1
    maxErrors     = '-1'

    withLabel:process_single {
        cpus   = { 1                   }
        memory = { 6.GB  * task.attempt }
        time   = { 4.h   * task.attempt }
        disk   = { 20.GB * task.attempt }
    }
    withLabel:process_low {
        cpus   = { 2     * task.attempt }
        memory = { 12.GB * task.attempt }
        time   = { 4.h   * task.attempt }
        disk   = { 20.GB * task.attempt }
    }
    withLabel:process_medium {
        cpus   = { 6     * task.attempt }
        memory = { 36.GB * task.attempt }
        time   = { 8.h   * task.attempt }
        disk   = { 40.GB * task.attempt }
    }
    withLabel:process_high {
        cpus   = { 12    * task.attempt }
        memory = { 72.GB * task.attempt }
        time   = { 16.h  * task.attempt }
        disk   = { 100.GB * task.attempt }
    }
    withLabel:process_high_memory {
        memory = { 200.GB * task.attempt }
        disk   = { 200.GB * task.attempt }
    }
}

Plugin initialization error on AWS HealthOmics

HealthOmics pre-installs fixed versions of the nf-schema, nf-core-utils, nf-prov, and nf-fgbio plugins per Nextflow version and ignores any other version declared in nextflow.config. If your pipeline (e.g., an nf-core pipeline) pins a different plugin version than HealthOmics provides, initialization fails. Update the pipeline's declared plugin version to match the one HealthOmics pins for its Nextflow version and re-register the workflow.

ECR access denied on AWS HealthOmics

If a run fails while pulling a container image, confirm the ECR repository's resource policy grants read access to both the Workbench pod account and the omics.amazonaws.com service principal, and that the repository is in the same AWS Region as your workspace. See Configure external ECR repository resources for more details.

Issues previewing HTML files

If you're trying to preview an HTML file that references images in other folders (e.g., a MultiQC report), you'll need to click Open in GCP. Select the HTML file in the GCP console to preview it.