Nextflow
Categories:
Prior reading: Overview of the Command Line Interface
Purpose: This document provides summary examples of commands that may be used alone or in combination when running Nextflow workflows through the Workbench CLI.
Prerequisites
These instructions assume that you have already installed the Workbench CLI, or are working in a cloud environment where it has been installed, and that you have logged in and identified the workspace you want to work with as described in the Basic Usage Examples.
Basic invocation
Run a Nextflow hello world example
terra nextflow run hello
This requires having the Docker image set and container running, or Nextflow installed locally. For Docker support, export TERRA_CLI_DOCKER_MODE=DOCKER_AVAILABLE
before installing terra
.
Running Nextflow from a workspace
Run an example Nextflow workflow in the context of the a workspace (i.e. in the workspace’s backing Google project). This is the same example workflow used in the GCLS tutorial.
Import the workflow code from GitHub
git clone https://github.com/nextflow-io/rnaseq-nf.git
cd rnaseq-nf
git checkout v2.0
cd ..
(Optional) Create a bucket in the workspace
terra resource create gcs-bucket --name=mybucket --bucket-name=mybucket
Customize the workflow configuration
Update the gls
section of the rnaseq-nf/nextflow.config
file to point to the workspace project and bucket we just created.
gls {
params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
params.multiqc = 'gs://rnaseq-nf/multiqc'
process.executor = 'google-lifesciences'
process.container = 'nextflow/rnaseq-nf:latest'
workDir = "$TERRA_mybucket/scratch"
google.region = 'us-east1'
google.project = "$GOOGLE_CLOUD_PROJECT"
google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
google.lifeSciences.network = 'network'
google.lifeSciences.subnetwork = 'subnetwork'
}
Do a dry-run
Confirm the config is set correctly.
terra nextflow config rnaseq-nf/main.nf -profile gls
Launch the workflow
Actually send the workflow out for execution. Note that this specific example takes about 10 minutes to run to completion.
terra nextflow run rnaseq-nf/main.nf -profile gls
(Optional) Send metrics to a Nextflow Tower server
To send metrics about the workflow run to a Nextflow Tower server, first define an environment variable with the Tower access token. Then specify the -with-tower
flag when kicking off the workflow.
export TOWER_ACCESS_TOKEN=*****
terra nextflow run hello -with-tower
terra nextflow run rnaseq-nf/main.nf -profile gls -with-tower
Other relevant operations
Call the gcloud CLI tools
This means that gcloud is configured with the backing Google project and environment variables are defined that contain workspace and resource properties (e.g. bucket names, pet service account email).
terra gcloud config get-value project
terra gsutil ls
terra bq version
ℹ️ terra gcloud
ℹ️ terra gsutil
ℹ️ terra bq
List supported third-party tools
The CLI runs these tools in a Docker image, if app-launch
mode is DOCKER_CONTAINER
. If the app-launch
mode is LOCAL_PROCESS
, the CLI will assume the tools are available in the current shell environment and launch them there.
terra app list
Print the tag of an image
Retrieve information about the specific version of the environment image.
terra config get image
Last Modified: 16 November 2023