Git integrations with cloud environments

Prior reading: Overview of Cloud Environments

Purpose: This document provides detailed instructions for working with Git and GitHub in Verily Workbench.

Introduction to working with Git in Verily Workbench

When researchers collaborate on the same Verily Workbench workspace, they may have one or more Git repositories where the shared notebook or scripts live. When researchers work from their local machine or Vertex AI notebook, a common workflow is that they clone the upstream git repositories, create a new branch, make some changes, push to the git repository for peer review, and then merge to the main branch.

You can add your Git repositories to a workspace to support a similar flow. Any Git repositories added to a given workspace before a cloud environment’s creation will automatically be cloned to that cloud environment. You can then use the git CLI or JupyterLab git extension to commit and push changes.

Add a Git repository to a workspace

When you add a reference to a git repository to your workspace, it will automatically be cloned to any cloud environment that you subsequently create in that workspace.

Adding a git repository to a workspace does NOT retroactively add that repository to any cloud environments that you had already created previously. If you want to retroactively add a later-defined GitHub reference to a cloud environment, you can run the following command in a Terminal window of the cloud enviroment, in the directory in which you want the repo to be located:

terra git clone --resource=<YOUR_GITHUB_RESOURCE_NAME>

Or, you can clone the repo using its URI directly via:

git clone <REPO_URI>

Through the web UI

To add a Git repository to your workspace through the web UI, go to the Environments tab in your workspace and click ‘Add repository’ in the ‘Git repositories’ box.

Through the CLI

To add a Git repository to your workspace through the CLI, use the terra resource add-ref git-repo command.

Here is an example command line that adds a reference to the DataBiosphere/terra-example-notebooks repository to the current workspace, under the name example_notebook:

terra resource add-ref git-repo --name="example_notebook" --repo-url=https://github.com/DataBiosphere/terra-example-notebooks.git

The Workbench CLI is installed for you in your cloud environments; or you can install it locally.

Choosing between the HTTPS or SSH form of the repository URI

To work with a public repository, use the https:// form of the repository’s URI. This will allow the repo to be cloned without specifying an SSH key.

To work with a private repository, use the git@ form of the repository’s URI. This is necessary for Workbench to succesfully clone the repository and requires an SSH key. See Set up an SSH key below for more details.

You can find the URI of your repository on its GitHub page; it should look like this:

Set up an SSH key

Working with private Git repositories requires that you set up an SSH key. The system will use this key to manage secure access to your repository from within your workspace.

Create an SSH key

Through the web UI

TODO: plug in the detailed instructions here (I can’t find anything that says how to do this the very first time – does it not require any user intervention?)

Through the CLI

To create a key using the CLI, run the following command:

terra user ssh-key generate

Optionally, you can run terra user ssh-key generate --save-to-file to save the SSH key pairs to your local machine.

Associate your public SSH key with your GitHub account

Once you’ve created an SSH key, copy the public SSH key string as shown below, then follow the GitHub instructions to associate the key with your GitHub account.

We recommend that you name the key as something like “Workbench SSH key” to distinguish it from your personal SSH keys.

If your GitHub organization sets up SAML Single Sign On (SSO), select Configure SSO on the Workbench SSH key that you added in Step 2 and select the organization to authorize your key for.

Manage your public SSH key

You can view, copy and regenerate your public SSH key at any time.

Keep in mind that when you regenerate your SSH key, you have to update it in GitHub. You will not be able to work with your private repositories in Workbench until you have done so. And conversely, GitHub will still accept the old key until you revoke it there, which is a problem if you think the old key might have been compromised. To address this, log in to your GitHub account and go to https://github.com/settings/keys. From there you can revoke your previously linked Workbench SSH key and attach the new key.

Through the web UI

1. Navigate to your user profile by clicking on your profile icon in the top right corner of any page in the web UI:

   

2. To copy your public SSH key, switch to ‘Groups and keys’ in the dialog that pops up, then click ‘Copy’:

To regenerate your public SSH key, click ‘Regenerate’ instead of ‘Copy’ in the step shown above.

Through the CLI

To manage your public SSH key through the CLI, use the terra user ssh-key commands.

If running any of these commands throws an error and you have never visited the web UI, you might not have a key. In that case, see the key creation instructions above.

Additional notes about SSH keys

• The procedure described above adds the SSH key to the ssh-agent in your workspace.

• The SSH key you set up as described above will also work to manage authentication to private repositories that you clone ‘manually’ into your cloud environment, i.e. without adding them to the workspace.

• There can only be one SSH key per Workbench user account. GitHub does not allow having one SSH key for multiple GitHub accounts, so we encourage you to work with a single GitHub account within a given cloud environment.
  If you must switch between different GitHub accounts (e.g. one for personal and one for business) within the same environment, you can upload a personal SSH key that is tied to your alternative GitHub account from your local computer, and add it to the VM SSH agent. This (unofficial, non-Verily) guide may help.