Workbench CLI commands
Categories:
Prior reading: Overview of the Command Line Interface
Purpose: This document provides a list of the Workbench CLI commands and information about their usage. For a full list of commands that includes details about subcommand arguments, see https://support.workbench.verily.com/docs/commands/wb/.
List available commands
Usage: wb [COMMAND]
Workbench CLI
Commands:
app Run applications in the workspace.
auth Commands to manage user credentials.
bq Call bq in the Workbench workspace.
cluster Use spark clusters in the workspace.
config Commands to configure the CLI.
pod, crg Manage Workbench pods.
cromwell Commands related to Cromwell workflows.
folder Commands related to folder.
gcloud Call gcloud in the Workbench workspace.
git Call git in the Workbench workspace.
group Manage groups of Workbench users.
gsutil Call gsutil in the Workbench workspace.
nextflow Call nextflow in the Workbench workspace.
notebook Use Notebooks in the workspace.
organization, org Manage workbench organizations.
resolve Resolve a resource to its cloud id or path.
resource Manage references and controlled resources in the workspace.
security Manage Workbench security credentials such as SSH keys.
server Connect to a Workbench server.
spend Manage spend profiles.
status Print details about the current workspace and server.
user Manage users.
version Get the installed version.
workspace Setup a Workbench workspace.
Standalone commands
The status command prints details about the current workspace and server.
ℹ️ wb status
The version command prints the installed version.
ℹ️ wb version
The gcloud, git, gsutil, bq, and nextflow commands call third-party
applications in the context of a Verily Workbench workspace.
ℹ️ wb gcloud
ℹ️ wb git
ℹ️ wb gsutil
ℹ️ wb bq
ℹ️ wb nextflow
The resolve command is an alias for the wb resource resolve command.
ℹ️ wb resolve
Groupings of sub-commands
The other commands are groupings of sub-commands, described in the sections below.
appApplicationsauthAuthenticationconfigConfigcromwellCromwellgitGitgroupGroupsgsutilgsutilnotebookNotebooksresourceResourcesserverServerspendSpenduserUserworkspaceWorkspace
Applications
Usage: wb app [COMMAND]
Run applications in the workspace.
Commands:
execute [FOR DEBUG] Execute a command in the application container for the
Workbench workspace, with no setup.
list List the supported applications.
ℹ️ wb app
The Workbench CLI allows running supported third-party tools within the context of a
workspace. To see supported tools, run wb app list.
The app-launch configuration property controls how tools are run (e.g., in a Docker
container or a local child process).
If you pass the --workspace flag, it must come immediately after the tool:
# Works
> wb bq --workspace=<workspace-id> ls
# Doesn't work, --workspace is passed to bq instead of Workbench
> wb bq ls --workspace=<workspace-id>
For creating resources such as a BigQuery dataset or GCS bucket, you must create them through Workbench rather than GCP tools. This is because Workbench configures permissions for you.
# Works
> wb resource create gcs-bucket --id=<resource-id>
# Doesn't work
> wb gsutil mb gs://<bucket-name>
Authentication
Usage: wb auth [COMMAND]
Retrieve and manage user credentials.
Commands:
login Authorize the CLI to access Workbench APIs and data with user credentials.
revoke Revoke credentials from an account.
status Print details about the currently authorized account.
ℹ️ wb auth
Only one user can be logged in at a time. Call wb auth login to log in as a
different user.
Login uses the Google OAuth 2.0 installed application flow.
You don’t need to log in again after switching workspaces. You will need to log in again after switching servers, because different Workbench deployments may have different OAuth flows.
By default, the CLI opens a browser window for the user to click through the
OAuth flow. For some use cases (e.g., Cloud Shell, cloud environment), this is not
practical because there is no default (or any) browser on the machine. The CLI
has a browser option that controls this
behavior. wb config set browser MANUAL means the user can copy the URL into
a browser on a different machine (e.g., their laptop), complete the login prompt,
and then copy/paste the response token back into a shell on the machine where
they want to use the Workbench CLI. For example:
> wb config set browser MANUAL
Browser launch mode for login is MANUAL (CHANGED).
> wb auth login
Please open the following address in a browser on any machine:
https://accounts.google.com/o/oauth2/auth?access_type=offline&approval_prompt=force&client_id=[...]
Please enter code: *****
Login successful: testuser@gmail.com
Config
Usage: wb config [COMMAND]
Configure the CLI.
Commands:
get Get a configuration property value.
list List all configuration properties and their values.
set Set a configuration property value.
ℹ️ wb config
These commands are property getters and setters for configuring the Workbench CLI. Currently the available configuration properties are:
OPTION VALUE DESCRIPTION
app-launch LOCAL_PROCESS app launch mode
browser AUTO browser launch for login
image us-east1-docker.pkg.dev/terra-vdevel-happy... docker image id
resource-limit 1000 max number of resources to allow per workspace
console-logging OFF logging level for printing directly to the terminal
file-logging INFO logging level for writing to files
server verily (unset)
workspace (unset) (unset)
format TEXT output format
wb-path /usr/local/bin/wb wb path
default-reosurce-id (unset) default resource id
aws-vault-path /usr/local/bin/aws-vault aws-vault path
cache-with-aws-vault false cache with aws-vault
Cromwell
Utility commands for using the Cromwell workflow engine with Workbench.
Usage: wb cromwell [COMMAND]
Commands related to Cromwell workflows.
Commands:
generate-config Generate cromwell.conf under the user-specified path.
ℹ️ wb cromwell
To use Cromwell to run a WDL workflow in a cloud environment:
- Run:
wb cromwell generate-config \ (--workspace-bucket-name=bucket_name | --google-bucket-name=gs://my-bucket) \ [--dir=my/path] - Either
workspace-bucket-nameorgoogle-bucket-nameis required to specify the bucket used by Cromwell for workflow orchestration.workspace-bucket-nameis a Workbench resource name.google-bucket-nameis a Google Cloud Storage bucket name. Ifgoogle-bucket-namedoes not begin with thegs://prefix, it will be automatically added.
- Run
java -Dconfig.file=path/to/cromwell.conf -jar cromwell/cromwell-81.jar server. This starts Cromwell server onlocalhost:8000. - In another terminal window, run
cromshell. Enterlocalhost:8000for cromwell server. - Start workflow through cromshell: e.g.,
cromshell submit workflow.wdl inputs.json [options.json] [dependencies.zip]
For more information, see here.
Git
Usage: wb git [COMMAND]
Call git command in the wb workspace. Besides calling normal Git operation, this command allow cloning git-repo resources in the workspace.
Commands:
all Clone all the git-repo resources in the workspace. Usage: wb git clone --all
resource Clone specified git-repo resources in the workspace. Usage: wb git clone --resource=<repoResource1Name> --resource=<repoResource2Name>
ℹ️ wb git
To add a Git repo:
> wb resource add-ref git-repo --id=<resource_id> --repo-url=<repo_url>
Groups
Usage: wb group [COMMAND]
Workbench contains a directory-like system for managing groups of users. These
commands are utility wrappers around the group management API endpoints.
Commands:
add-user Add a user to a group with a given policy.
create Create a new Workbench group.
delete Delete an existing Workbench group.
describe Describe the group.
list List the groups to which the current user belongs.
list-users List the users in a group.
remove-user Remove a user from a group with a given policy.
ℹ️ wb group
Workbench groups are managed by SAM. These commands are utility wrappers around the group endpoints.
Say a Workbench group’s email is mygroup@mydomain.com. The option for the --name flag is mygroup,
not mygroup@mydomain.com:
> wb group list-users --name=mygroup
Adding a member to a Workbench group implicitly adds their pet service accounts. For example, say wb-user is added to mygroup@mydomain.com. When mygroup is granted access to a resource, wb-user is able to access that resource from any of their workspaces.
gsutil
You can run wb gsutil or wb gcloud alpha storage. gcloud alpha storage
is a newer version of gsutil. It doesn’t support everything, but what it does
support may be significantly faster.
ℹ️ wb gsutil
Notebooks
Usage: wb notebook [COMMAND]
Commands to manage Notebook resources within the workspace.
Commands:
start Start a stopped Notebook instance within your workspace.
stop Stop a running Notebook instance within your workspace.
ℹ️ wb notebook
You can create
a GCP Notebook
controlled resource
with wb resource create gcp-notebook --id=<resourceid> [--workspace=<id>]
. These stop and start
commands are provided for convenience. You can also stop and start the notebook
using the gcloud notebooks instances start/stop commands.
Resources
Usage: wb resource [COMMAND]
Manage references and controlled resources in the workspace.
Commands:
add-ref, add-referenced Add a new referenced resource.
check-access Check if you have access to a referenced resource.
create, create-controlled Add a new controlled resource.
credentials Retrieve temporary credentials to access a cloud resource.
delete Delete a resource from the workspace.
describe Describe a resource.
list List all resources.
list-tree List all resources and folders in tree view.
mount Mounts all workspace bucket resources.
move Move resource to a folder.
open-console Retrieve console link to access a cloud resource.
resolve Resolve a resource to its cloud id or path.
unmount Unmounts all workspace bucket resources.
update Update the properties of a resource.
ℹ️ wb resource
A controlled resource is a cloud resource
that is managed by Workbench. It exists within the current workspace context (e.g., a bucket
within the workspace Google project). You can create these with the create command.
A referenced resource is a cloud resource
that is NOT managed by Workbench. It exists outside the current workspace context (e.g., a
BigQuery dataset hosted outside of Workbench or in another workspace). You can add these with the
add-ref command. The workspace currently supports the following referenced resources:
gcs-bucketgcs-objectbq-datasetbq-tablegit-repo
The check-access command lets you see whether you have access to a particular
resource. This is useful when a different user created or added the resource and
subsequently shared the workspace with you. check-access currently always
returns true for git-repo reference type because workspaces don’t support
authentication to external Git services yet.
The list of resources in a workspace is maintained
on the Workbench Workspace Manager server. The CLI caches this list of resources locally.
Third-party tools can access resource details via environment variables (e.g., $WORKBENCH_mybucket
holds the gs:// URL of the workspace bucket resource named mybucket). The
CLI updates the cache on every call to a wb resource command. If you
are working in a shared workspace, you can run wb resource list (for
example) to pick up any changes that your collaborators have made.
GCS bucket lifecycle rules
GCS bucket lifecycle rules are specified by passing a JSON-formatted file path
to the
wb resource create gcs-bucket command. The expected JSON structure matches
the one used by the gsutil lifecycle
command. This
structure is a subset of the GCS
resource specification
. Below are some example file contents for specifying a lifecycle rule.
(1) Change the storage class to ARCHIVE after 10 days.
{
"rule": [
{
"action": {
"type": "SetStorageClass",
"storageClass": "ARCHIVE"
},
"condition": {
"age": 10
}
}
]
}
(2) Delete any objects with storage class STANDARD that were created before
December 3, 2007.
{
"rule": [
{
"action": {
"type": "Delete"
},
"condition": {
"createdBefore": "2007-12-03",
"matchesStorageClass": ["STANDARD"]
}
}
]
}
(3) Delete any objects that are more than 365 days old.
{
"rule": [
{
"action": {
"type": "Delete"
},
"condition": {
"age": 365
}
}
]
}
There is also a command shortcut for specifying this type of lifecycle rule in example 3:
wb resource create gcs-bucket --id=<mybucket> --bucket-name=<mybucket> --auto-delete=365
GCS bucket object reference
A reference to a GCS bucket object can be created by calling
wb resource add-ref gcs-object --id=<referenceid> --bucket-name=<mybucket> --object-name=<myobject>
Reference to a file or folder
A file or folder is treated as an object in GCS bucket. By either creating a
folder through the cloud console UI or copying an existing folder of files to
the GCS bucket, a user can create a folder or file object. The user can then create a
reference to the folder/file if they have at least READER access to the bucket
and/or READER access to the folder/file.
Reference to multiple objects under a folder
Different from other referenced resource type, there is also support for
creating a reference to objects in the folder. For instance, a user may create a
a foo/ folder with bar.txt and secret.txt in it. If the user has at least
READER access to foo/ folder, they have access to anything in the foo/ folder. So
they can add a reference to foo/bar.txt, foo/\* or foo/\*.txt.
NOTE Be careful to provide the correct object name when creating a reference. We only check if the user has
READERaccess to the provided path; we do not check whether the object exists. This is helpful because when referencing to foo/*, it is actually not a real object! Therefore, a reference tofooo/(where objectfooodoes not exist) can be created if the user hasREADERaccess to the bucket orfoo/\*.png(where there is no png files) if they have access to thefoo/folder.
Update a reference resource
Users can update the name and description of a reference resource. Users can also
update a reference resource to another of the same type. For instance, if a user
creates a reference resource to BigQuery dataset foo and later on wants to point to
BigQuery dataset bar in the same project, one can use
wb resource update --id=<fooReferenceId> --new-dataset-id=bar to update
the reference. However, users are not allowed to update the reference to a
different type (e.g., updating a dataset reference to a data table reference is not
allowed).
Mounting workspace resources
Users can mount GCS buckets and referenced folder objects locally to the user’s
home directory in $HOME/workspace/ by running wb resource mount.
Users can specify the --id flag with the ID of a GCS bucket or GCS object
resource to only mount that individual resource. This flag is useful for
remounting a resource that had failed to mount or has been moved to a different
folder in the workspace.
By default, controlled GCS buckets and referenced folder objects created by the
user will be mounted with read-write permissions while controlled buckets
created by other users and referenced bucket folders will be mounted with
read-only permissions. Users can override this default behavior by specifying
the --read-only flag. Ex: wb resource mount --read-only for all mounts to
be read-only or wb resource mount --id=mybucket --read-only=false for all
mounts to be read-write.
Users can specify the --disable-cache flag to disable file
metadata caching and file type caching for objects in the mounted buckets. List
operations such as ls will be slower, but will reflect the most up-to-date
state of the bucket. This is useful when working with collaborators in a shared
workspace. See more details in
the gcsfuse
repository.
Mount failures
If a mount has failed, an empty directory will be left at mount point with the
resource name and a suffix error string indicating the failure. Users can
remount the bucket with wb resource mount after resolving bucket access or bucket reference issues.
Unmounting a single resource can fail if the resource has been renamed or moved
to a different workspace folder. In this case, users can either
run wb resource unmount to unmount all mounted resources
in $HOME/workspace/. Alternatively, users can directly list out all mounted file systems
with mount and then unmount the resource using its mount path
with fusermount -u (for Linux) or umount (for MacOS).
Server
Usage: wb server [COMMAND]
Connect to a Workbench server.
Commands:
list List all available Workbench servers.
set Set the Workbench server to connect to.
status Print status and details of the Workbench server context.
ℹ️ wb server
A Workbench server or environment is a set of connected Workbench services (e.g., Workspace Manager, Data Repo, SAM).
Workspaces exist on a single server, so switching servers will change the list of workspaces available to you.
Spend
Usage: wb spend [COMMAND]
Manage spend profiles.
Commands:
create-profile Create a spend profile.
enable Enable use of a spend profile for a user or group.
delete-profile Delete a spend profile.
disable Disable use of a spend profile for a user or group.
list-users List the users enabled on a spend profile.
These commands are intended for admin users. See ADMIN.md for more details.
User
Usage: wb user [COMMAND]
Manage users.
Commands:
invite Invite a new user.
status Check the registration status of a user.
ssh-key
Ensure you have the latest CLI version. To install a new CLI version, first manually uninstall the existing CLI via
rm -R ~/.workbench, and then install the latest CLI.
wb user ssh-key helps Workbench support source control in a notebook environment. It handles the ssh key of the current user. There is one Workbench SSH key
per user. With this SSH key, you can perform
source control in a Workbench-managed notebook instance using Git.
To set up an SSH key, run wb user ssh-key add to add the Workbench SSH key
to your local machine. You should see in the output an SSH public key starting
with ssh-rsa. Copy the public key from the command output and
add it to GitHub, as per these instructions.
If you think your key is compromised (e.g., the private key on your local machine
is leaked to another user), you must delete the key from your GitHub account and
run wb user ssh-key generate to generate a new Workbench SSH key. Once a new
key is generated, you need to
associate this new key with your GitHub account again.
Workspace
Usage: wb workspace [COMMAND]
Set up a Workbench workspace.
Commands:
add-user Add a user or group to the workspace.
break-glass Grant break-glass access to a workspace user.
configure-aws Generate an AWS configuration file for a workspace.
create Create a new workspace.
delete Delete an existing workspace.
delete-property Delete the workspace properties.
describe Describe the existing workspace.
duplicate Duplicate an existing workspace
list List all workspaces the current user can access.
list-users List the users of the workspace.
remove-user Remove a user or group from the workspace.
set Set the workspace to an existing one.
set-property Set the workspace properties.
update Update an existing workspace.
ℹ️ wb workspace
A Workbench workspace is backed by a Google project. Creating/deleting a workspace also creates/deletes the project.
The break-glass command is intended for admin users. See ADMIN.md
for more details.
Folder
Usage: wb folder [COMMAND]
Commands related to folder.
Commands:
tree Show the folder hierarchy.
set-property Set the folder properties.
create Create a folder.
delete Delete a folder.
update Update a folder.
ℹ️ wb folder
Last Modified: 12 May 2024