Getting started with Nextflow on Verily Workbench

Introduction

Nextflow is a framework for creating data-driven computational pipelines. It allows you to create scalable and reproducible scientific workflows using software containers.

On Verily Workbench, the easiest way to get started using nextflow is via a cloud environment, where nextflow and the Workbench CLI are already installed for you. However, you can also use a local installation of the Workbench CLI.

To get set up, you will first need to create a Workbench workspace (if need be), and then create some resources associated with that workspace. Then (optional but recommended), spin up a cloud environment on which to run nextflow.

The following sections walk you through that setup, then show how terra makes it easy to configure and run a nextflow pipeline.

Create a workspace

If you don’t already have a Workbench workspace that you want to use, you can create one via either the Workbench CLI or the web UI.

• Using the Web UI
• Using the CLI

terra status
terra auth login  # if need be
terra workspace list

terra workspace create –name=<workspace-name>

terra workspace set –id=<workspaceid>

Create some workspace resources

If you haven’t already, you’ll need to create a GCS bucket resource, which will be used by Nextflow for staging and logging.

We’ll also create a Git resource that points to the nextflow example repo. Any notebook instances that you subsequently create in your workspace will automatically clone that repo for you.

• Using the Web UI
• Using the CLI

terra resource create gcs-bucket --name=ws_files --bucket-name=${GOOGLE_CLOUD_PROJECT}-ws-files \
  --description="Bucket for reports and provenance records."

terra resource add-ref git-repo --name=rnaseq-nf-repo --repo-url=https://github.com/nextflow-io/rnaseq-nf.git

terra resource list

Create a cloud environment to run Nextflow (recommended)

Next, create a Workbench cloud environment on which to run the nextflow example.

Your cloud environment will have nextflow pre-installed, and any workspace git repo resources— such as the one we just defined— will be automatically cloned. However, if you want to run the example on your local machine, you can install nextflow yourself.

• Using the Web UI
• Using the CLI

terra resource create gcp-notebook --name=<notebook_resource_name> \
  --description=<description>

terra resource describe --name <notebook_resource_name>

Configure and run the example nextflow pipeline

Because you created a git repo resource, the rnaseq-nf example should be automatically cloned into your new cloud environment, and you should see its directory, rnaseq-nf, at the top level of your file system. (If it is not there, you can run:
terra git clone --resource=rnaseq-nf-repo).

In the JupyterLab Terminal window, change to the rnaseq-nf directory and check out the tagged v2.1 version.

cd rnaseq-nf
git checkout v2.1

Then, still in the rnaseq-nf directory, edit the nextflow.config file as follows. Replace the gls entry with the following snippet. Edit the workDir line to replace <your_bucket_resource_name> with the name of the GCS bucket resource you created.

As you’ll see below, you will run the nextflow pipeline via terra, and terra will substitute the correct value for the environment variables in the config before it runs.

 gls {
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'
    process.executor = 'google-lifesciences'
    process.container = 'nextflow/rnaseq-nf:latest'
    // edit the following line for your bucket resource
    workDir = "$TERRA_<your_bucket_resource_name>/nf"
    google.location = 'us-central1'
    google.region  = 'us-central1'
    google.project = "$GOOGLE_CLOUD_PROJECT"
    google.lifeSciences.network = 'network'
    google.lifeSciences.subnetwork = 'subnetwork'
    google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
  }

If you’ve forgotten the name of the bucket resource you created, you can find it via:

terra resource list  # this shows the resource names

Then, change to the parent directory of rnaseq-nf and sanity-check your config changes.

cd
terra nextflow config rnaseq-nf/main.nf -profile gls

You should see output that shows instantiated values for your workspace project, GCS bucket, and service account email.

Run the Nextflow example workflow via terra

After you check your config, you’re ready to run the nextflow example. In the parent directory of rnaseq-nf, run:

terra nextflow run rnaseq-nf/main.nf -profile gls

The workflow will take about 10 minutes to complete.