Nextflow

Prior reading: Overview of the Command Line Interface

Purpose: This document provides summary examples of commands that may be used alone or in combination when running Nextflow workflows through the Workbench CLI.

Prerequisites

These instructions assume that you have already installed the Workbench CLI, or are working in a cloud environment where it has been installed, and that you have logged in and identified the workspace you want to work with as described in the Basic Usage Examples.

Basic invocation

Run a Nextflow “Hello World” example:

wb nextflow run hello

ℹ️ wb nextflow run

This requires having the Docker image set and container running, or Nextflow installed locally. For Docker support, run export WORKBENCH_CLI_DOCKER_MODE=DOCKER_AVAILABLE before installing wb.

Running Nextflow from a workspace

Run an example Nextflow workflow in the context of a workspace (i.e. in the workspace’s underlying Google project). This is the same example workflow used in the GCLS tutorial.

Import the workflow code from GitHub

    git clone https://github.com/nextflow-io/rnaseq-nf.git
    cd rnaseq-nf
    git checkout v2.0
    cd ..

(Optional) Create a bucket in the workspace

Replace <mybucket> with your ID and bucket name values.

wb resource create gcs-bucket --id=<mybucket> --bucket-name=<mybucket>

ℹ️ wb resource create

Customize the workflow configuration

Update the gls section of the rnaseq-nf/nextflow.config file to point to the workspace project and bucket we just created.

gls {
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'
    process.executor = 'google-lifesciences'
    process.container = 'nextflow/rnaseq-nf:latest'
    workDir = "$WORKBENCH_mybucket/scratch"

    google.region  = 'us-east1'
    google.project = "$GOOGLE_CLOUD_PROJECT"

    google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
    google.lifeSciences.network = 'network'
    google.lifeSciences.subnetwork = 'subnetwork'
}

Do a dry run

Confirm the config is set correctly.

wb nextflow config rnaseq-nf/main.nf -profile gls

ℹ️ wb nextflow config

Launch the workflow

Actually send the workflow out for execution. Note that this specific example takes about 10 minutes to run to completion.

wb nextflow run rnaseq-nf/main.nf -profile gls

ℹ️ wb nextflow run

(Optional) Send metrics to a Nextflow Tower server

To send metrics about the workflow run to a Nextflow Tower server, first define an environment variable with the Tower access token. Then specify the -with-tower flag when kicking off the workflow.

export TOWER_ACCESS_TOKEN=*****
wb nextflow run hello -with-tower
wb nextflow run rnaseq-nf/main.nf -profile gls -with-tower

ℹ️ wb nextflow run

Other relevant operations

Call the gcloud CLI tools

This means that gcloud is configured with the underlying Google project and environment variables are defined that contain workspace and resource properties (e.g., bucket names, pet service account email).

wb gcloud config get-value project
wb gsutil ls
wb bq version

ℹ️ wb gcloud ℹ️ wb gsutil ℹ️ wb bq

List supported third-party tools

The CLI runs these tools in a Docker image, if app-launch mode is DOCKER_CONTAINER. If the app-launch mode is LOCAL_PROCESS, the CLI will assume the tools are available in the current shell environment and launch them there.

wb app list

ℹ️ wb app list

Print the tag of an image

Retrieve information about the specific version of the environment image.

wb config get image

ℹ️ wb config get