Skip to main content
On Lightdash, we generate some files like:
  • Images for Slack unfurl and scheduled deliveries
  • Results on JSONL/CSV/Excel format
These files need to be stored in a S3 Compatible Cloud storage. Some options are GCP Buckets, S3 Storage and MinIO.
S3-compatible storage is required. Lightdash will fail to start if S3_ENDPOINT, S3_BUCKET, and S3_REGION are not set. All warehouses (including Snowflake) fetch async query results from S3.

​
Configure cloud storage on Google Cloud Platform

In the Google Cloud console, go to Cloud Storage and create a bucket with these details:
  • Give it a unique Bucket name, such as lightdash-cloud-file-storage-eu
  • Select a region, such as multi-region US, multi-region EU, or a single region like europe-west1
  • Select the default Standard storage class
  • Enable Enforce public access prevention on this bucket
  • Select Uniform access control
  • Set Protection to none
Set S3_ENDPOINT to https://storage.googleapis.com and S3_REGION to auto. Then choose how Lightdash authenticates to the bucket. Google Cloud Storage accepts two methods, and they are not interchangeable.

​
Option A: HMAC keys

Use this option anywhere outside Google Kubernetes Engine (GKE), or when you want the simplest setup.
  1. Go to Settings > Interoperability and create an Access key for service account.
  2. Copy the access key and the secret.
  3. Set S3_ACCESS_KEY and S3_SECRET_KEY.
These keys are long-lived static secrets. Rotate them on a schedule. Google Cloud Storage supports several active keys per service account, so you can rotate without downtime.

​
Option B: Workload identity, with no keys

Use this option when Lightdash runs on GKE and you do not want static secrets in the cluster. Lightdash sends a short-lived Google OAuth token instead of signing requests with a key. Set S3_AUTH_MODE=gcp_oauth and leave S3_ACCESS_KEY and S3_SECRET_KEY unset.
S3_USE_CREDENTIALS_FROM does not enable this option, even though its ecs and ec2 values look like they should. Every source that setting accepts is an AWS credential provider, and Google Cloud Storage cannot verify AWS credentials. On GKE, use S3_AUTH_MODE=gcp_oauth.
Grant these permissions to the Google service account that the pod runs as:
  1. Bind the Kubernetes service account to the Google service account, as described in the GKE workload identity documentation.
  2. Grant the roles/storage.objectUser role on the bucket. Do not grant a project-wide role such as Editor.
  3. Grant the roles/iam.serviceAccountTokenCreator role on the service account itself. The member and the resource are the same service account. Lightdash needs this role to sign download URLs.
  4. Enable the iamcredentials.googleapis.com API in the project.
If you skip step 3, the deployment looks healthy. Uploads, downloads, and result caching all work. Only signed download links fail, and only when a user asks for one. The error is Permission 'iam.serviceAccounts.signBlob' denied on resource.
Set these environment variables:
Both URL styles work with Google Cloud Storage. Path style also works for bucket names that contain dots, which break TLS certificate matching in the other style. Signed URLs in this mode last at most 7 days, which is a Google Cloud Storage limit. Lightdash fails to start if S3_EXPIRATION_TIME exceeds 604800 seconds. To develop locally against a bucket that uses this mode, run:
Your own user credentials cannot sign URLs, and the error is Cannot sign data without 'client_email'. Impersonating the service account fixes it. Do not download a service account key file.

​
Configure cloud storage on AWS

  • Navigate to the S3 section of the AWS Management Console and click on the Create Bucket button.
  • Give your bucket a name and select the region where you want to store your data.
  • Next, you need to set the permissions for your bucket. Make it private.
To export your S3 credentials, you need to follow these steps:
  • Navigate to the IAM section and click on the Users tab.
  • Click on the user whose credentials you want to export.
  • Click on the Security Credentials tab and locate the Access Keys section.
  • Click on the Create Access Key button.
  • Download the CSV file that contains your Access Key ID and Secret Access Key.
Check this guide to see what’s the right S3_ENDPOINT for your bucket

​
Configure cloud storage using MinIO

Creating a bucket in MinIO
  • Login to the MinIO console and click on “Buckets” in the side bar
  • Click on “Create Bucket”
  • Give your bucket a name and click “Create Bucket”
Creating access credentials in MinIO
  • Click on “Access Keys” in the side bar
  • Click “Create access key”
  • Give a name to your new access key and click “Create”
  • Download the JSON file containing both your Access Key ID and Secret Access Key
MinIO needs path style bucket URLs, for this you will need to set S3_FORCE_PATH_STYLE: true in your environment variables.

​
Azure Storage

Azure Blob Storage is not natively compatible with the S3 API. While Lightdash supports external object storage by allowing integration with S3-compatible APIs, Azure’s storage service does not provide this compatibility out of the box. This means that you cannot use Azure Blob Storage as a drop-in replacement for S3 in Lightdash deployments. Instead, you can use one of the following S3-compatible solutions within your Azure setup:
  • MinIO: S3-compatible object storage
  • s3proxy: A lightweight proxy that adds an S3-compatible API layer on top of Azure Blob Storage.

​
Configure Lightdash to use S3 credentials

To enable Lightdash to use your S3 bucket for cloud storage, you’ll need to set the following environment variables:
For a comprehensive list of all possible S3-related environment variables and other configurations, please visit the Environment Variables documentation.

​
Use separate storage for query results

Query results use the main S3 configuration by default. To store them with a different S3-compatible provider, set the RESULTS_S3_* variables that differ from it — credentials and path-style addressing have RESULTS_S3_* equivalents too:
Any omitted result storage setting inherits its corresponding S3_* value.
Raw JSONL result downloads use a presigned URL that browsers fetch directly from RESULTS_S3_ENDPOINT. This endpoint must be reachable from users’ browsers; S3_PUBLIC_ENDPOINT does not apply to the query results bucket. The bucket can remain private, but an IP policy that limits GetObject to Lightdash’s backend egress addresses will block these downloads.

​
Authenticate without static keys

How you do this depends on the cloud provider, because the two mechanisms are unrelated. On AWS. Omit S3_ACCESS_KEY and S3_SECRET_KEY, and the AWS SDK looks for an IAM role automatically. To control which credential sources it tries and in what order, set S3_USE_CREDENTIALS_FROM. For more detail, read the AWS SDK for JavaScript documentation on setting credentials in Node.js. On Google Cloud. Set S3_AUTH_MODE=gcp_oauth. See Option B: workload identity, with no keys. S3_USE_CREDENTIALS_FROM does not work here, because every source it accepts is an AWS credential provider. Signed URLs last at most 7 days in both cases, whatever you set S3_EXPIRATION_TIME to. The limit comes from the signing algorithm, not from Lightdash.

​
Production bucket strategy

For an enterprise deployment, use a dedicated bucket per purpose, each with a scoped credential that can only touch its own bucket. This is how Lightdash Cloud sets up storage:
  • Results / cache bucket — add a lifecycle rule deleting objects after 1 day. Results are ephemeral, so this keeps the bucket small and cheap.
  • Pre-aggregations bucket (if using pre-aggregates) — lifecycle delete after ~30 days.
  • Data-apps bucket (if using data apps) — persistent, with no delete lifecycle.
Block all public access on every bucket, and prefer short-lived credentials over static keys. On EKS, use IAM roles for service accounts (IRSA). On GKE, set S3_AUTH_MODE=gcp_oauth to use workload identity. In both cases, signed URLs last at most 7 days. If users share download links that must outlive the signed-URL TTL (S3_EXPIRATION_TIME, default 3 days), enable PERSISTENT_DOWNLOAD_URLS_ENABLED: "true" — see persistent download URLs.