Nextflow

Prior reading: Command-line interface overview

Purpose: This document provides summary examples of commands that may be used alone or in combination when running Nextflow workflows through the Workbench CLI.

Prerequisites

These instructions assume that you have already installed the Workbench CLI (command-line interface), or are working in a cloud app where it has been installed, and that you have logged in and identified the workspace you want to work with as described in the Basic Usage Examples.

Basic invocation

Run a Nextflow "Hello World" example:

wb nextflow run hello

ℹ️ wb nextflow run

This requires having the Docker image set and container running, or Nextflow installed locally. For Docker support, run export WORKBENCH_CLI_DOCKER_MODE=DOCKER_AVAILABLE before installing wb.

Running Nextflow from a workspace

Run an example Nextflow workflow in the context of a workspace (i.e. in the workspace's underlying Google project). This is the same example workflow used in the GCLS tutorial.

Import the workflow code from GitHub

    git clone https://github.com/nextflow-io/rnaseq-nf.git
    cd rnaseq-nf
    git checkout v2.0
    cd ..

(Optional) Create a bucket in the workspace

Replace <mybucket> with your ID and bucket name values.

wb resource create gcs-bucket --id=<mybucket> --bucket-name=<mybucket>

ℹ️ wb resource create

Customize the workflow configuration

You can determine how a workflow is run by specifying the executor. The configuration for two executors are given below.

Google Cloud Life Sciences API (deprecated)

Update the gls section of the rnaseq-nf/nextflow.config file to point to the workspace project and bucket we just created.

gls {
    // Workflow params
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'

    // Google Life Sciences config
    process.executor = 'google-lifesciences'
    process.container = 'nextflow/rnaseq-nf:latest'
    workDir = "$WORKBENCH_mybucket/scratch"

    google.region  = 'us-east1'
    google.project = "$GOOGLE_CLOUD_PROJECT"

    google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
    google.lifeSciences.network = 'network'
    google.lifeSciences.subnetwork = 'subnetwork'
}

Google Batch API

Update the google-batch section of the rnaseq-nf/nextflow.config file to point to the workspace project and bucket we just created.

'google-batch' {
    // Workflow params
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'

    // Google Batch config
    process.executor = 'google-batch'
    process.container = 'nextflow/rnaseq-nf:latest'
    workDir = "$WORKBENCH_mybucket/scratch"

    google.region  = 'us-central1'
    google.project = "$GOOGLE_CLOUD_PROJECT"

    google.batch.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
    google.batch.usePrivateAddress = true
    google.batch.network = 'global/networks/network'
    google.batch.subnetwork = 'regions/us-east1/subnetworks/subnetwork'
}

Do a dry run

Confirm the config is set correctly. Choose the appropriate job executor for the profile. The following command selects Google Batch API. For Cloud Life Sciences API, specify -profile gls.

wb nextflow config rnaseq-nf/main.nf -profile google-batch

ℹ️ wb nextflow config

Launch the workflow

Actually send the workflow out for execution. Note that this specific example takes about 10 minutes to run to completion.

wb nextflow run rnaseq-nf/main.nf -profile google-batch

ℹ️ wb nextflow run

(Optional) Send metrics to a Nextflow Tower server

To send metrics about the workflow run to a Nextflow Tower server, first define an environment variable with the Tower access token. Then specify the -with-tower flag when kicking off the workflow.

export TOWER_ACCESS_TOKEN=*****
wb nextflow run hello -with-tower
wb nextflow run rnaseq-nf/main.nf -profile google-batch -with-tower

ℹ️ wb nextflow run

Other relevant operations

Call the gcloud CLI tools

This means that gcloud is configured with the underlying Google project and environment variables are defined that contain workspace and resource properties (e.g., bucket names, workspace service account email).

wb gcloud config get-value project
wb gsutil ls
wb bq version

You can also list Batch API jobs.

wb gcloud batch jobs list

ℹ️ wb gcloud ℹ️ wb gsutil ℹ️ wb bq

List supported third-party tools

The CLI runs these tools in a Docker image, if app-launch mode is DOCKER_CONTAINER. If the app-launch mode is LOCAL_PROCESS, the CLI will assume the tools are available in the current shell environment and launch them there.

wb utility list

ℹ️ wb utility list

Print the tag of an image

Retrieve information about the specific version of the app image.

wb config get image

ℹ️ wb config get