Workbench CLI commands

Detailed explanations about the Workbench CLI commands and their options

Prior reading: Overview of the Command Line Interface

Purpose: This document provides a list of the Workbench CLI commands and information about their usage. For a full list of commands that includes details about subcommand arguments, see https://support.workbench.verily.com/docs/commands/wb/.



List available commands

Usage: wb [COMMAND]
Workbench CLI
Commands:
  app                             Run applications in the workspace.
  auth                            Commands to manage user credentials.
  bq                              Call bq in the Workbench workspace.
  cluster                         Use spark clusters in the workspace.
  config                          Commands to configure the CLI.
  pod, crg                        Manage Workbench pods.
  cromwell                        Commands related to Cromwell workflows.
  folder                          Commands related to folder.
  gcloud                          Call gcloud in the Workbench workspace.
  git                             Call git in the Workbench workspace.
  group                           Manage groups of Workbench users.
  gsutil                          Call gsutil in the Workbench workspace.
  nextflow                        Call nextflow in the Workbench workspace.
  notebook                        Use Notebooks in the workspace.
  organization, org               Manage workbench organizations.
  resolve                         Resolve a resource to its cloud id or path.
  resource                        Manage references and controlled resources in the workspace.
  security                        Manage Workbench security credentials such as SSH keys.
  server                          Connect to a Workbench server.
  spend                           Manage spend profiles.
  status                          Print details about the current workspace and server.
  user                            Manage users.
  version                         Get the installed version.
  workspace                       Setup a Workbench workspace.

Standalone commands

The status command prints details about the current workspace and server.

ℹ️ wb status

The version command prints the installed version.

ℹ️ wb version

The gcloud, git, gsutil, bq, and nextflow commands call third-party applications in the context of a Verily Workbench workspace.

ℹ️ wb gcloud ℹ️ wb git ℹ️ wb gsutil ℹ️ wb bq ℹ️ wb nextflow

The resolve command is an alias for the wb resource resolve command.

ℹ️ wb resolve

Groupings of sub-commands

The other commands are groupings of sub-commands, described in the sections below.

Applications

Usage: wb app [COMMAND]
Run applications in the workspace.
Commands:
  execute  [FOR DEBUG] Execute a command in the application container for the
             Workbench workspace, with no setup.
  list     List the supported applications.

ℹ️ wb app

The Workbench CLI allows running supported third-party tools within the context of a workspace. To see supported tools, run wb app list.

The app-launch configuration property controls how tools are run (e.g., in a Docker container or a local child process).

If you pass the --workspace flag, it must come immediately after the tool:

# Works
> wb bq --workspace=<workspace-id> ls

# Doesn't work, --workspace is passed to bq instead of Workbench
> wb bq ls --workspace=<workspace-id>

For creating resources such as a BigQuery dataset or GCS bucket, you must create them through Workbench rather than GCP tools. This is because Workbench configures permissions for you.

# Works
> wb resource create gcs-bucket --id=<resource-id>

# Doesn't work
> wb gsutil mb gs://<bucket-name>

Authentication

Usage: wb auth [COMMAND]
Retrieve and manage user credentials.
Commands:
  login   Authorize the CLI to access Workbench APIs and data with user credentials.
  revoke  Revoke credentials from an account.
  status  Print details about the currently authorized account.

ℹ️ wb auth

Only one user can be logged in at a time. Call wb auth login to log in as a different user.

Login uses the Google OAuth 2.0 installed application flow.

You don’t need to log in again after switching workspaces. You will need to log in again after switching servers, because different Workbench deployments may have different OAuth flows.

By default, the CLI opens a browser window for the user to click through the OAuth flow. For some use cases (e.g., Cloud Shell, cloud environment), this is not practical because there is no default (or any) browser on the machine. The CLI has a browser option that controls this behavior. wb config set browser MANUAL means the user can copy the URL into a browser on a different machine (e.g., their laptop), complete the login prompt, and then copy/paste the response token back into a shell on the machine where they want to use the Workbench CLI. For example:

> wb config set browser MANUAL
Browser launch mode for login is MANUAL (CHANGED).

> wb auth login
Please open the following address in a browser on any machine:
  https://accounts.google.com/o/oauth2/auth?access_type=offline&approval_prompt=force&client_id=[...]
Please enter code: *****
Login successful: testuser@gmail.com

Config

Usage: wb config [COMMAND]
Configure the CLI.
Commands:
  get   Get a configuration property value.
  list  List all configuration properties and their values.
  set   Set a configuration property value.

ℹ️ wb config

These commands are property getters and setters for configuring the Workbench CLI. Currently the available configuration properties are:

OPTION                VALUE                                          DESCRIPTION
app-launch            LOCAL_PROCESS                                  app launch mode
browser               AUTO                                           browser launch for login
image                 us-east1-docker.pkg.dev/terra-vdevel-happy...  docker image id
resource-limit        1000                                           max number of resources to allow per workspace
console-logging       OFF                                            logging level for printing directly to the terminal
file-logging          INFO                                           logging level for writing to files
server                verily                                         (unset)
workspace             (unset)                                        (unset)
format                TEXT                                           output format
wb-path               /usr/local/bin/wb                              wb path
default-reosurce-id   (unset)                                        default resource id
aws-vault-path        /usr/local/bin/aws-vault                       aws-vault path
cache-with-aws-vault  false                                          cache with aws-vault

Cromwell

Utility commands for using the Cromwell workflow engine with Workbench.

Usage: wb cromwell [COMMAND]
Commands related to Cromwell workflows.
Commands:
  generate-config  Generate cromwell.conf under the user-specified path.

ℹ️ wb cromwell

To use Cromwell to run a WDL workflow in a cloud environment:

  • Run:
    wb cromwell generate-config \
        (--workspace-bucket-name=bucket_name | --google-bucket-name=gs://my-bucket) \
        [--dir=my/path]
    
  • Either workspace-bucket-name or google-bucket-name is required to specify the bucket used by Cromwell for workflow orchestration.
    • workspace-bucket-name is a Workbench resource name.
    • google-bucket-name is a Google Cloud Storage bucket name. If google-bucket-name does not begin with the gs:// prefix, it will be automatically added.
  • Run java -Dconfig.file=path/to/cromwell.conf -jar cromwell/cromwell-81.jar server. This starts Cromwell server on localhost:8000.
  • In another terminal window, run cromshell. Enter localhost:8000 for cromwell server.
  • Start workflow through cromshell: e.g., cromshell submit workflow.wdl inputs.json [options.json] [dependencies.zip]

For more information, see here.

Git

Usage: wb git [COMMAND]
Call git command in the wb workspace. Besides calling normal Git operation, this command allow cloning git-repo resources in the workspace.
Commands:
  all        Clone all the git-repo resources in the workspace. Usage: wb git clone --all
  resource   Clone specified git-repo resources in the workspace. Usage: wb git clone --resource=<repoResource1Name> --resource=<repoResource2Name>

ℹ️ wb git

To add a Git repo:

> wb resource add-ref git-repo --id=<resource_id> --repo-url=<repo_url>

Groups

Usage: wb group [COMMAND]
Workbench contains a directory-like system for managing groups of users. These
commands are utility wrappers around the group management API endpoints.
Commands:
  add-user     Add a user to a group with a given policy.
  create       Create a new Workbench group.
  delete       Delete an existing Workbench group.
  describe     Describe the group.
  list         List the groups to which the current user belongs.
  list-users   List the users in a group.
  remove-user  Remove a user from a group with a given policy.

ℹ️ wb group

Workbench groups are managed by SAM. These commands are utility wrappers around the group endpoints.

Say a Workbench group’s email is mygroup@mydomain.com. The option for the --name flag is mygroup, not mygroup@mydomain.com:

> wb group list-users --name=mygroup

Adding a member to a Workbench group implicitly adds their pet service accounts. For example, say wb-user is added to mygroup@mydomain.com. When mygroup is granted access to a resource, wb-user is able to access that resource from any of their workspaces.

gsutil

You can run wb gsutil or wb gcloud alpha storage. gcloud alpha storage is a newer version of gsutil. It doesn’t support everything, but what it does support may be significantly faster.

ℹ️ wb gsutil

Notebooks

Usage: wb notebook [COMMAND]
Commands to manage Notebook resources within the workspace.
Commands:
  start  Start a stopped Notebook instance within your workspace.
  stop   Stop a running Notebook instance within your workspace.

ℹ️ wb notebook

You can create a GCP Notebook controlled resource with wb resource create gcp-notebook --id=<resourceid> [--workspace=<id>] . These stop and start commands are provided for convenience. You can also stop and start the notebook using the gcloud notebooks instances start/stop commands.

Resources

Usage: wb resource [COMMAND]
Manage references and controlled resources in the workspace.
Commands:
  add-ref, add-referenced    Add a new referenced resource.
  check-access               Check if you have access to a referenced resource.
  create, create-controlled  Add a new controlled resource.
  credentials                Retrieve temporary credentials to access a cloud resource.
  delete                     Delete a resource from the workspace.
  describe                   Describe a resource.
  list                       List all resources.
  list-tree                  List all resources and folders in tree view.
  mount                      Mounts all workspace bucket resources.
  move                       Move resource to a folder.
  open-console               Retrieve console link to access a cloud resource.
  resolve                    Resolve a resource to its cloud id or path.
  unmount                    Unmounts all workspace bucket resources.
  update                     Update the properties of a resource.

ℹ️ wb resource

A controlled resource is a cloud resource that is managed by Workbench. It exists within the current workspace context (e.g., a bucket within the workspace Google project). You can create these with the create command.

A referenced resource is a cloud resource that is NOT managed by Workbench. It exists outside the current workspace context (e.g., a BigQuery dataset hosted outside of Workbench or in another workspace). You can add these with the add-ref command. The workspace currently supports the following referenced resources:

  • gcs-bucket
  • gcs-object
  • bq-dataset
  • bq-table
  • git-repo

The check-access command lets you see whether you have access to a particular resource. This is useful when a different user created or added the resource and subsequently shared the workspace with you. check-access currently always returns true for git-repo reference type because workspaces don’t support authentication to external Git services yet.

The list of resources in a workspace is maintained on the Workbench Workspace Manager server. The CLI caches this list of resources locally. Third-party tools can access resource details via environment variables (e.g., $WORKBENCH_mybucket holds the gs:// URL of the workspace bucket resource named mybucket). The CLI updates the cache on every call to a wb resource command. If you are working in a shared workspace, you can run wb resource list (for example) to pick up any changes that your collaborators have made.

GCS bucket lifecycle rules

GCS bucket lifecycle rules are specified by passing a JSON-formatted file path to the wb resource create gcs-bucket command. The expected JSON structure matches the one used by the gsutil lifecycle command. This structure is a subset of the GCS resource specification . Below are some example file contents for specifying a lifecycle rule.

(1) Change the storage class to ARCHIVE after 10 days.

{
  "rule": [
    {
      "action": {
        "type": "SetStorageClass",
        "storageClass": "ARCHIVE"
      },
      "condition": {
        "age": 10
      }
    }
  ]
}

(2) Delete any objects with storage class STANDARD that were created before December 3, 2007.

{
  "rule": [
    {
      "action": {
        "type": "Delete"
      },
      "condition": {
        "createdBefore": "2007-12-03",
        "matchesStorageClass": ["STANDARD"]
      }
    }
  ]
}

(3) Delete any objects that are more than 365 days old.

{
  "rule": [
    {
      "action": {
        "type": "Delete"
      },
      "condition": {
        "age": 365
      }
    }
  ]
}

There is also a command shortcut for specifying this type of lifecycle rule in example 3:

wb resource create gcs-bucket --id=<mybucket> --bucket-name=<mybucket> --auto-delete=365

GCS bucket object reference

A reference to a GCS bucket object can be created by calling

wb resource add-ref gcs-object --id=<referenceid> --bucket-name=<mybucket> --object-name=<myobject>
Reference to a file or folder

A file or folder is treated as an object in GCS bucket. By either creating a folder through the cloud console UI or copying an existing folder of files to the GCS bucket, a user can create a folder or file object. The user can then create a reference to the folder/file if they have at least READER access to the bucket and/or READER access to the folder/file.

Reference to multiple objects under a folder

Different from other referenced resource type, there is also support for creating a reference to objects in the folder. For instance, a user may create a a foo/ folder with bar.txt and secret.txt in it. If the user has at least READER access to foo/ folder, they have access to anything in the foo/ folder. So they can add a reference to foo/bar.txt, foo/\* or foo/\*.txt.

NOTE Be careful to provide the correct object name when creating a reference. We only check if the user has READER access to the provided path; we do not check whether the object exists. This is helpful because when referencing to foo/*, it is actually not a real object! Therefore, a reference to fooo/ (where object fooo does not exist) can be created if the user has READER access to the bucket or foo/\*.png (where there is no png files) if they have access to the foo/ folder.

Update a reference resource

Users can update the name and description of a reference resource. Users can also update a reference resource to another of the same type. For instance, if a user creates a reference resource to BigQuery dataset foo and later on wants to point to BigQuery dataset bar in the same project, one can use wb resource update --id=<fooReferenceId> --new-dataset-id=bar to update the reference. However, users are not allowed to update the reference to a different type (e.g., updating a dataset reference to a data table reference is not allowed).

Mounting workspace resources

Users can mount GCS buckets and referenced folder objects locally to the user’s home directory in $HOME/workspace/ by running wb resource mount.

Users can specify the --id flag with the ID of a GCS bucket or GCS object resource to only mount that individual resource. This flag is useful for remounting a resource that had failed to mount or has been moved to a different folder in the workspace.

By default, controlled GCS buckets and referenced folder objects created by the user will be mounted with read-write permissions while controlled buckets created by other users and referenced bucket folders will be mounted with read-only permissions. Users can override this default behavior by specifying the --read-only flag. Ex: wb resource mount --read-only for all mounts to be read-only or wb resource mount --id=mybucket --read-only=false for all mounts to be read-write.

Users can specify the --disable-cache flag to disable file metadata caching and file type caching for objects in the mounted buckets. List operations such as ls will be slower, but will reflect the most up-to-date state of the bucket. This is useful when working with collaborators in a shared workspace. See more details in the gcsfuse repository.

Mount failures

If a mount has failed, an empty directory will be left at mount point with the resource name and a suffix error string indicating the failure. Users can remount the bucket with wb resource mount after resolving bucket access or bucket reference issues.

Unmounting a single resource can fail if the resource has been renamed or moved to a different workspace folder. In this case, users can either run wb resource unmount to unmount all mounted resources in $HOME/workspace/. Alternatively, users can directly list out all mounted file systems with mount and then unmount the resource using its mount path with fusermount -u (for Linux) or umount (for MacOS).

Server

Usage: wb server [COMMAND]
Connect to a Workbench server.
Commands:
  list    List all available Workbench servers.
  set     Set the Workbench server to connect to.
  status  Print status and details of the Workbench server context.

ℹ️ wb server

A Workbench server or environment is a set of connected Workbench services (e.g., Workspace Manager, Data Repo, SAM).

Workspaces exist on a single server, so switching servers will change the list of workspaces available to you.

Spend

Usage: wb spend [COMMAND]
Manage spend profiles.
Commands:
  create-profile    Create a spend profile.
  enable            Enable use of a spend profile for a user or group.
  delete-profile    Delete a spend profile.
  disable           Disable use of a spend profile for a user or group.
  list-users        List the users enabled on a spend profile.            

These commands are intended for admin users. See ADMIN.md for more details.

User

Usage: wb user [COMMAND]
Manage users.
Commands:
  invite    Invite a new user.
  status    Check the registration status of a user.         

ssh-key

Ensure you have the latest CLI version. To install a new CLI version, first manually uninstall the existing CLI via rm -R ~/.workbench, and then install the latest CLI.

wb user ssh-key helps Workbench support source control in a notebook environment. It handles the ssh key of the current user. There is one Workbench SSH key per user. With this SSH key, you can perform source control in a Workbench-managed notebook instance using Git.

To set up an SSH key, run wb user ssh-key add to add the Workbench SSH key to your local machine. You should see in the output an SSH public key starting with ssh-rsa. Copy the public key from the command output and add it to GitHub, as per these instructions.

If you think your key is compromised (e.g., the private key on your local machine is leaked to another user), you must delete the key from your GitHub account and run wb user ssh-key generate to generate a new Workbench SSH key. Once a new key is generated, you need to associate this new key with your GitHub account again.

Workspace

Usage: wb workspace [COMMAND]
Set up a Workbench workspace.
Commands:
  add-user         Add a user or group to the workspace.
  break-glass      Grant break-glass access to a workspace user.
  configure-aws    Generate an AWS configuration file for a workspace.
  create           Create a new workspace.
  delete           Delete an existing workspace.
  delete-property  Delete the workspace properties.
  describe         Describe the existing workspace.
  duplicate        Duplicate an existing workspace
  list             List all workspaces the current user can access.
  list-users       List the users of the workspace.
  remove-user      Remove a user or group from the workspace.
  set              Set the workspace to an existing one.
  set-property     Set the workspace properties.
  update           Update an existing workspace.

ℹ️ wb workspace

A Workbench workspace is backed by a Google project. Creating/deleting a workspace also creates/deletes the project.

The break-glass command is intended for admin users. See ADMIN.md for more details.

Folder

Usage: wb folder [COMMAND]
Commands related to folder.
Commands:
  tree          Show the folder hierarchy.
  set-property  Set the folder properties.
  create        Create a folder.
  delete        Delete a folder.
  update        Update a folder.

ℹ️ wb folder

Last Modified: 21 May 2024