Introducing Supabase Pipelines

Today we're introducing Supabase Pipelines: managed change-data-capture pipelines that replicate your Postgres tables to analytical destinations in near real time.

Pipelines reads changes from your Postgres database and writes them to external destinations. It uses logical replication to capture inserts, updates, deletes, and truncates as they happen. Setup takes minutes in the Supabase Dashboard.

The first supported destination is Google BigQuery.

Pipelines is powered by the open-source Supabase ETL engine. You can find the code on GitHub at github.com/supabase/etl.

Why separate OLTP and OLAP?#

Postgres is excellent for transactional workloads like reading a single user record or inserting an order. But when you need to scan millions of rows for analytics, Postgres slows down.

Analytical systems are designed for this. They can aggregate massive datasets orders of magnitude faster, compress data more efficiently, and handle complex analytical queries that would choke a transactional database.

Pipelines gives you the best of both worlds: keep your app fast on Postgres while unlocking analytics on purpose-built systems.

How it works#

Pipelines captures every change in your Postgres database and delivers it to your analytics destination in near real time.

Here's how:

1. You create a Postgres publication that defines which tables to replicate

2. You add a destination that connects the publication to an analytical system

3. The pipeline reads changes from the publication through a logical replication slot

4. Changes are batched and written to your destination

5. Your data is available for querying in the destination

The pipeline starts with an initial copy of your selected tables, then switches to streaming mode. Your analytics stay fresh with latency measured in milliseconds to seconds.

Setting up Pipelines#

You configure Pipelines entirely through the Supabase Dashboard. No code required.

Step 1: Create a publication#

A publication defines which tables to replicate. You create it with SQL or via the UI:

Step 2: Enable Pipelines#

Navigate to Database in your Supabase Dashboard. Select the Replication tab, click Add destination, choose a Pipelines destination, and click Enable Pipelines.

Step 3: Configure the destination#

Configure your destination details, choose a publication, and click Create and start pipeline.

Step 4: Monitor your pipeline#

The Dashboard shows pipeline status and lag. You can start, stop, restart, or delete pipelines from the actions menu.

Available destinations#

Our goal with Pipelines is to let you connect your existing data systems to Supabase. We're actively expanding the list of supported destinations. The first supported destination is Google BigQuery.

BigQuery is Google's serverless data warehouse, built for large-scale analytics. It handles petabytes of data and integrates well with existing BI tools and data pipelines.

When you replicate to BigQuery, Pipelines handles the ongoing change delivery so your analytical system stays up to date.

Adding and removing tables#

You can modify which tables are replicated after your pipeline is running.

To add a table:

To remove a table:

After changing your publication, restart the pipeline from the Dashboard actions menu for the changes to take effect.

Note: The pipeline does not remove data from your destination when you remove a table from a publication. This is by design to prevent accidental data loss.

When to use Pipelines vs read replicas#

Read replicas and Pipelines solve different problems.

Read replicas help when you need to scale concurrent queries, but they're still Postgres. They don't make analytics faster.

Pipelines moves your data to systems built for analytics. You get faster queries on large datasets, lower storage costs through compression, and complete separation between your production workload and analytics.

You can use both: read replicas for application read scaling, Pipelines for analytics.

Things to know#

Replication with Pipelines has a few constraints to be aware of:

• Tables must have primary keys (this is a Postgres logical replication requirement)
• Generated columns are not supported
• Custom data types are replicated as strings
• Schema changes are not automatically propagated to destinations
• Data is replicated as-is, without transformation
• During the initial copy phase, changes accumulate in the WAL and are replayed once streaming begins

We're working on schema change support and additional destinations, and evaluating different streaming techniques to improve flexibility and performance.

Pricing#

Pipelines pricing is still being finalized during alpha. You can request access now; we'll announce pricing later and notify you before any charges apply.

Get started#

Pipelines is currently in alpha. Alpha features can be unstable and may introduce breaking changes while we evaluate the product direction, refine the feature set, and incorporate customer feedback. To request access, contact your account manager or fill out the form in the Dashboard.

If you want to dive into the code, the underlying Supabase ETL engine is open source and written in Rust. Check out the repository at github.com/supabase/etl.

Note: Supabase ETL is now Supabase Pipelines. This video was recorded before the rename; Supabase ETL now refers to the underlying engine that powers Pipelines.