Git integrations with cloud environments

Prior reading: Cloud environments overview

Purpose: This document provides detailed instructions for working with Git and GitHub in Verily Workbench.

Introduction

When researchers collaborate on the same Verily Workbench workspace, they may have one or more Git repositories where the shared notebook or scripts live. When researchers work from their local machine or Vertex AI notebook, a common workflow is cloning the upstream Git repositories, creating a new branch, making some changes, pushing to the Git repository for peer review, and then merging to the main branch.

You can add your Git repositories to a workspace to support a similar flow. Any Git repositories added to a given workspace before a cloud environment’s creation will automatically be cloned to that cloud environment. You can then use the Git CLI or JupyterLab Git extension to commit and push changes.

Add a Git repository to a workspace

When you add a reference to a Git repository to your workspace, it will automatically be cloned to any cloud environment that you subsequently create in that workspace.

Adding a Git repository to a workspace does NOT retroactively add that repository to any cloud environments that you had already created previously. If you want to retroactively add a later-defined GitHub reference to a cloud environment, you can run the following command in a Terminal window of the cloud enviroment, in the directory in which you want the repo to be located:

wb git clone --resource=<YOUR_GITHUB_RESOURCE_NAME>

Or, you can clone the repo using its URI directly via:

git clone <REPO_URI>

Through the web UI

To add a Git repository to your workspace through the web UI, go to the Environments tab in your workspace and click + Add repository in the Git repositories box.

Through the CLI

To add a Git repository to your workspace through the CLI, use the wb resource add-ref git-repo command.

Here’s an example command line that adds a reference to the DataBiosphere/terra-example-notebooks repository to the current workspace, under the name example_notebook:

wb resource add-ref git-repo --name="example_notebook" --repo-url=https://github.com/DataBiosphere/terra-example-notebooks.git

The Workbench CLI is installed for you in your cloud environments; or you can install it locally.

Choosing between the HTTPS or SSH form of the repository URI

To work with a public repository, use the https:// form of the repository’s URI. This will allow the repo to be cloned without specifying an SSH key.

To work with a private repository, use the git@ form of the repository’s URI. This is necessary for Workbench to succesfully clone the repository and requires an SSH key. See Set up an SSH key below for more details.

You can find the URI of your repository on its GitHub page:

Set up an SSH key

Create an SSH key

Through the web UI

An SSH key should be auto-generated the first time you access the web UI. Click your user profile icon in the top right corner, then click Your profile. Click on the Groups & keys tab to view your public SSH key.

Through the CLI

To view your SSH key using the CLI, run the following command:

wb security ssh-key get

If you receive an error and you have never visited the web UI, you may not have a key. To create one, run:

wb security ssh-key generate

Optionally, you can run wb security ssh-key generate --save-to-file to save the SSH key pairs to your local machine.

Associate your public SSH key with your GitHub account

The public SSH key is not automatically tied to your GitHub account. After you’ve generated your SSH key, you’ll need to add the public key to your GitHub account. Copy the public SSH key string as shown below, then follow the GitHub instructions to associate the key with your GitHub account.

We recommend that you name the key as something like “Workbench SSH key” to distinguish it from your personal SSH keys.

If your GitHub organization sets up SAML Single Sign On (SSO), select Configure SSO on the Workbench SSH key that you added in Step 8 of the GitHub instructions and select the organization to authorize your key for.

Manage your public SSH key

You can view, copy, and regenerate your public SSH key at any time.

Keep in mind that when you regenerate your SSH key, you have to update it in GitHub. You won’t be able to work with your private repositories in Workbench until you have done so. And conversely, GitHub will still accept the old key until you revoke it there, which is a problem if you think the old key might have been compromised. To address this, log in to your GitHub account and go to https://github.com/settings/keys. From there you can revoke your previously linked Workbench SSH key and attach the new key.

Through the web UI

1. Navigate to your user profile by clicking on your name in the bottom left corner of any page in the web UI. Click Your profile:

   

2. To copy your public SSH key, switch to Groups & keys in the dialog that pops up, then click Copy:

To regenerate your public SSH key, click Regenerate instead of Copy in the step shown above.

Through the CLI

To manage your public SSH key through the CLI, use the wb security ssh-key commands.

To view your SSH key, run:

wb security ssh-key get

To regenerate an SSH key, run:

wb security ssh-key generate

If running any of these commands throws an error and you have never visited the web UI, you might not have a key. In that case, see the key creation instructions above.

Working on a local machine

Run the following commands only if you’re not on a cloud environment and you don’t have an SSH key set up on your local machine.

wb security ssh-key generate --save-to-file
mv vwb_id_rsa ~/.ssh
mv vwb_id_rsa ~/.ssh.pub
eval "$(ssh-agent -s)"
open ~/.ssh/config

In the config file, add:

Host *
  AddKeysToAgent yes
  IgnoreUnknown UseKeychain
  UseKeychain yes
  IdentityFile ~/.ssh/vwb_id_rsa

Additional notes about SSH keys

• The procedure described above adds the SSH key to the ssh-agent in your workspace.

• The SSH key you set up as described above will also work to manage authentication to private repositories that you clone manually into your cloud environment, i.e., without adding them to the workspace.

• There can only be one SSH key per Workbench user account. GitHub does not allow having one SSH key for multiple GitHub accounts, so we encourage you to work with a single GitHub account within a given cloud environment.
  If you must switch between different GitHub accounts (e.g., one for personal and one for business) within the same environment, you can upload a personal SSH key that is tied to your alternative GitHub account from your local computer, and add it to the VM SSH agent. This (unofficial, non-Verily) guide may help.