Nextflow

Prior reading: Overview of the Command Line Interface

Purpose: This document provides summary examples of commands that may be used alone or in combination when running Nextflow workflows through the Workbench CLI.

Prerequisites

These instructions assume that you have already installed the Workbench CLI, or are working in a cloud environment where it has been installed, and that you have logged in and identified the workspace you want to work with as described in the Basic Usage Examples.

Basic invocation

Run a Nextflow hello world example

terra nextflow run hello

ℹ️ terra nextflow run

This requires having the Docker image set and container running, or Nextflow installed locally. For Docker support, export TERRA_CLI_DOCKER_MODE=DOCKER_AVAILABLE before installing terra.

Running Nextflow from a workspace

Run an example Nextflow workflow in the context of the a workspace (i.e. in the workspace’s backing Google project). This is the same example workflow used in the GCLS tutorial.

Import the workflow code from GitHub

    git clone https://github.com/nextflow-io/rnaseq-nf.git
    cd rnaseq-nf
    git checkout v2.0
    cd ..

(Optional) Create a bucket in the workspace

terra resource create gcs-bucket --name=mybucket --bucket-name=mybucket

ℹ️ terra resource create

Customize the workflow configuration

Update the gls section of the rnaseq-nf/nextflow.config file to point to the workspace project and bucket we just created.

gls {
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'
    process.executor = 'google-lifesciences'
    process.container = 'nextflow/rnaseq-nf:latest'
    workDir = "$TERRA_mybucket/scratch"

    google.region  = 'us-east1'
    google.project = "$GOOGLE_CLOUD_PROJECT"

    google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
    google.lifeSciences.network = 'network'
    google.lifeSciences.subnetwork = 'subnetwork'
}

Do a dry-run

Confirm the config is set correctly.

terra nextflow config rnaseq-nf/main.nf -profile gls

ℹ️ terra nextflow config

Launch the workflow

Actually send the workflow out for execution. Note that this specific example takes about 10 minutes to run to completion.

terra nextflow run rnaseq-nf/main.nf -profile gls

ℹ️ terra nextflow run

(Optional) Send metrics to a Nextflow Tower server

To send metrics about the workflow run to a Nextflow Tower server, first define an environment variable with the Tower access token. Then specify the -with-tower flag when kicking off the workflow.

export TOWER_ACCESS_TOKEN=*****
terra nextflow run hello -with-tower
terra nextflow run rnaseq-nf/main.nf -profile gls -with-tower

ℹ️ terra nextflow run

Other relevant operations

Call the gcloud CLI tools

This means that gcloud is configured with the backing Google project and environment variables are defined that contain workspace and resource properties (e.g. bucket names, pet service account email).

terra gcloud config get-value project
terra gsutil ls
terra bq version

ℹ️ terra gcloud ℹ️ terra gsutil ℹ️ terra bq

List supported third-party tools

The CLI runs these tools in a Docker image, if app-launch mode is DOCKER_CONTAINER. If the app-launch mode is LOCAL_PROCESS, the CLI will assume the tools are available in the current shell environment and launch them there.

terra app list

ℹ️ terra app list

Print the tag of an image

Retrieve information about the specific version of the environment image.

terra config get image

ℹ️ terra config get