Nextflow

Prior reading: Command-line interface overview

Purpose: This document provides summary examples of commands that may be used alone or in combination when running Nextflow workflows through the Workbench CLI.

Prerequisites

These instructions assume that you have already installed the Workbench CLI (command-line interface), or are working in a cloud app where it has been installed, and that you have logged in and identified a GCP-backed workspace you want to work with as described in Basic usage.

Basic invocation

Run a Nextflow "Hello World" example:

wb nextflow run hello

ℹ️ wb nextflow run

This requires having the Docker image set and container running, or Nextflow installed locally. For Docker support, run export WORKBENCH_CLI_DOCKER_MODE=DOCKER_AVAILABLE before installing wb.

Running Nextflow from a workspace

Run an example Nextflow workflow in the context of a workspace (i.e., in the workspace's underlying Google project).

Import the workflow code from GitHub

    git clone https://github.com/nextflow-io/rnaseq-nf.git
    cd rnaseq-nf
    git checkout v2.0
    cd ..

Create a bucket in the workspace

Create a Cloud Storage bucket that will be used by Nextflow for task-level workflow logs and outputs. Replace <mybucket> with your ID and bucket name values.

wb resource create gcs-bucket --id=<mybucket> --bucket-name=<mybucket>

ℹ️ wb resource create

Customize the workflow configuration

You can determine how a workflow is run by specifying the executor. We'll use the Google Batch API executor.

Add or update the following google-batch profile to the profiles section of the rnaseq-nf/nextflow.config file.

Important configuration notes:

• The google.location is the region where Google Batch submits jobs and must match your workspace region (it defaults to us-central1 if unset) The Nextflow job location must be the same as the workspace region, or the job will fail.
• The google.batch.subnetwork must be in the same region as your VM and must use the format regions/<region>/subnetworks/subnetwork.
• The environment variables ($WORKBENCH_mybucket, $GOOGLE_CLOUD_PROJECT, $GOOGLE_SERVICE_ACCOUNT_EMAIL) are automatically set by the wb CLI and will be available in all wb-launched subprocesses. You don't need to define them manually in your terminal.
• Replace mybucket in $WORKBENCH_mybucket with your actual bucket name (the name you used when creating the bucket above).
• You can verify these environment variables are set correctly by running: wb utility execute env.
• You can save the container's runnable logs (regular .stdout, .stderr, and .log files, plus additional information from the batch runner) by specifying a logsPath value that points to a specific folder within a GCP bucket. You'll need to create the bucket's folder in advance. Note that a forward slash must follow the folder name.

'google-batch' {
    // Workflow params
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'

    // Google Batch config
    process.executor = 'google-batch'
    process.container = 'nextflow/rnaseq-nf:latest'
    // Replace 'mybucket' with your actual bucket name from the previous step
    workDir = '$WORKBENCH_mybucket/scratch'
    logsPath = '$WORKBENCH_mybucket/logs/'

    // Must match your VM's region
    google.location = 'us-central1'

    google.project = "$GOOGLE_CLOUD_PROJECT"

    google.batch.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
    google.batch.usePrivateAddress = true
    google.batch.network = 'global/networks/network'
    // Must match your VM's region
    google.batch.subnetwork = 'regions/us-central1/subnetworks/subnetwork'
}

Do a dry run

Confirm the config is set correctly. Choose the appropriate job executor for the profile.

wb nextflow config rnaseq-nf/main.nf -profile google-batch

ℹ️ wb nextflow config

Launch the workflow

Execute the workflow. Note that this specific example takes about 10 minutes to run to completion.

wb nextflow run rnaseq-nf/main.nf -profile google-batch

ℹ️ wb nextflow run

Troubleshooting workflow failures

If your workflow fails, you can diagnose the issue using multiple sources.

Google Cloud Console (recommended first step):

The Batch API in Google Cloud Console provide error messages for configuration issues:

1. Navigate to the Batch jobs page in your workspace's Google Cloud project
2. Click on your job to view its details
3. Click on the Events tab to see failure messages and error details

Common errors:

• Region mismatch: If you see errors about resources not being found or permission denied, check that your google.location, and google.batch.subnetwork all use the same region as your workspace. Region mismatches are a common cause of job failures.
• Subnet not found: Verify the google.batch.subnetwork format matches regions/<your-region>/subnetworks/subnetwork

Local logs:

• Check the .nextflow.log file in your current directory for detailed execution logs

• Run your workflow with the -with-report flag to generate an HTML report:

  wb nextflow run rnaseq-nf/main.nf -profile google-batch -with-report run-report.html
  

  This creates a run-report.html file with detailed execution statistics and error information

Verify environment variables:

If you suspect environment variable issues, check that wb has set them correctly:

wb utility execute env | grep -E 'WORKBENCH|GOOGLE'

This will show all workspace-related environment variables that Nextflow will use.

(Optional) Send metrics to a Nextflow Tower server

To send metrics about the workflow run to a Nextflow Tower server, first define an environment variable with the Tower access token. Then specify the -with-tower flag when kicking off the workflow.

export TOWER_ACCESS_TOKEN=*****
wb nextflow run hello -with-tower
wb nextflow run rnaseq-nf/main.nf -profile google-batch -with-tower

ℹ️ wb nextflow run

Other relevant operations

Call the gcloud CLI tools

This means that gcloud is configured with the underlying Google project and environment variables are defined that contain workspace and resource properties (e.g., bucket names, workspace service account email).

wb gcloud config get-value project
wb gcloud storage ls
wb bq version

You can also list Batch API jobs.

wb gcloud batch jobs list

ℹ️ wb gcloud ℹ️ wb bq

List supported third-party tools

The CLI runs these tools in a Docker image, if app-launch mode is DOCKER_CONTAINER. If the app-launch mode is LOCAL_PROCESS, the CLI will assume the tools are available in the current shell environment and launch them there.

wb utility list

ℹ️ wb utility list

Print the tag of an image

Retrieve information about the specific version of the app image.

wb config get image

ℹ️ wb config get