How to reference and work with Requester Pays buckets

Prior reading: Cloud cost management

Purpose: This document provides information about working with Requester Pays buckets in Verily Workbench.

Introduction

Access to data in Cloud Storage may incur cloud charges. Google Cloud does not charge users for accessing data in Cloud Storage as long as that data stays within its storage region. For example, moving data from a bucket in us-central1 to a cloud environment in us-central1 is free. Charges are incurred for moving data out of the storage region.

When a researcher copies data out of a Cloud Storage region, including downloading the data, the bucket owner is charged for all data transfer costs associated with the bucket. However, buckets can be configured to be a Requester Pays bucket. Requester Pays allows you to control Cloud Storage billing by shifting access costs to the users who request your data. For more on cloud charges and management see Cloud cost management.

Adding and browsing a reference to a Requester Pays bucket

Adding a data reference via the Web UI or CLI is described in How to work with data. When a Requester Pays bucket is added through the UI, an info message about costs related to the Requester Pays bucket is shown.

As you browse a Requester Pays bucket, costs for reading and downloading data are billed to the workspace’s Pod. Learn more in Cloud Storage Documentation.

Mounting a Requester Pays bucket

In a cloud environment, buckets mount on each startup. Mounting buckets set to Requester Pays may incur cost. See more about mounted workspace buckets in Accessing workspace files and folders from your Cloud Environment.

Accessing a Requester Pays bucket using gsutil or gcloud storage

To use gsutil to access a Requester Pays bucket, use the -u flag and supply a user project to be billed.

gsutil -u PROJECT_ID ls gs://BUCKET_NAME

To use gcloud storage to access a Requester Pays bucket, use the --billing-project flag:

gcloud storage ls gs://BUCKET_NAME --billing-project PROJECT_ID