Working with workspace context environment variables

Prior reading: Overview of the Command Line Interface

Purpose: This document explains what the workspace context is, and how to use it via the Workbench CLI.

What is the workspace context?

The Workbench CLI defines a workspace context for applications to run in.

The workspace context includes the following environment variables:

• GOOGLE_CLOUD_PROJECT: The ID of the Google project backing the workspace.
• GOOGLE_SERVICE_ACCOUNT_EMAIL: The email of the current user’s pet service account in the current workspace.
• Environment variables that are the name of the workspace resources, prefixed with TERRA_ are set to the resolved cloud identifier for those resources. For example, for a storage bucket called mybucket referenced in the workspace, there will be an environment variable called TERRA_mybucket pointing to the bucket’s cloud path, gs://mybucket. This applies to both referenced and controlled resources.

Referring to workspace context variables

Reference in a CLI command

To use a workspace reference in a Workbench CLI command, escape the environment variable to bypass the shell substitution on the host machine.

The following example shows the commands for creating a new controlled bucket resource and then using gsutil to get its IAM bindings:

> terra resource create gcs-bucket --name=mybucket --bucket_name=mybucket
Successfully added controlled GCS bucket.

> terra gsutil iam get \$TERRA_mybucket
  Setting the gcloud project to the workspace project
  Updated property [core/project].

  {
    "bindings": [
      {
        "members": [
          "projectEditor:terra-wsm-dev-e3d8e1f5",
          "projectOwner:terra-wsm-dev-e3d8e1f5"
        ],
        "role": "roles/storage.legacyBucketOwner"
      },
      {
        "members": [
          "projectViewer:terra-wsm-dev-e3d8e1f5"
        ],
        "role": "roles/storage.legacyBucketReader"
      }
    ],
    "etag": "CAE="
  }

Reference in a file

To use a workspace reference in a file or configuration that will be read by an application, do not escape the environment variable. Since this will be running inside the Docker container or local process, there is no need to bypass shell substitution.

The following example shows a Nextflow worfklow configuration file that includes a reference to a bucket resource in the workspace, the backing Google project, and the workspace pet service account email:

profiles {
  gls {
      params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
      params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
      params.multiqc = 'gs://rnaseq-nf/multiqc'
      process.executor = 'google-lifesciences'
      process.container = 'nextflow/rnaseq-nf:latest'
      workDir = "$TERRA_mybucket/scratch"

      google.region  = 'us-east1'
      google.project = "$GOOGLE_CLOUD_PROJECT"

      google.lifeSciences.serviceAccountEmail = "$GOOGLE_SERVICE_ACCOUNT_EMAIL"
      google.lifeSciences.network = 'network'
      google.lifeSciences.subnetwork = 'subnetwork'
  }
}

Execution environment

See all environment variables

Run terra app execute env to see all environment variables defined in the Docker container or local process when applications are launched.

The terra app execute ... command is primarily intended for debugging. It lets you execute any command in the Docker container or local process, not just the ones that are officially supported (i.e. gsutil, bq, gcloud, nextflow, git, which you can list by running terra app list).

Run unsupported tools

To run tools that are not yet supported by the Workbench CLI, or to use local versions of tools, set the app-launch configuration property to launch a child process on the local machine instead of inside a Docker container.

terra config set app-launch LOCAL_PROCESS

Then call the tool with terra app execute. Before running the tool command, the CLI defines environment variables for each workspace resource and configures gcloud with the workspace project. After running the tool command, the CLI restores the original gcloud project configuration.

terra app execute dsub \
    --provider google-v2 \
    --project \$GOOGLE_CLOUD_PROJECT \
    --regions us-central1 \
    --logging \$TERRA_MY_BUCKET/logging/ \
    --output OUT=\$TERRA_MY_BUCKET/output/out.txt \
    --command 'echo "Hello World" > "${OUT}"' \
    --wait

(Note: The command above came from the dsub README.)