Set up external replication

Configure publications and destinations for external replication.

External replication is powered by Supabase ETL and uses Postgres logical replication to stream changes from your database to external data systems. It provides a managed interface through the Dashboard to configure and monitor replication pipelines.

Setup overview#

External replication requires two main components: a Postgres publication (defines what to replicate) and a destination (where data is sent). Follow these steps to set up your replication pipeline.

Step 1: Create a Postgres publication#

A Postgres publication defines which tables and change types will be replicated from your database. You create publications using SQL.

Creating a publication#

The following SQL examples assume you have users and orders tables in your database.

Publication for specific tables

1
-- Create publication for both tables
2
create publication pub_users_orders
3
for table users, orders;

This publication tracks all changes (INSERT, UPDATE, DELETE, TRUNCATE) for both the users and orders tables.

Publication for all tables in a schema

1
-- Create a publication for all tables in the public schema
2
create publication pub_all_public for tables in schema public;

This tracks changes for all existing and future tables in the public schema.

Publication for all tables

1
-- Create a publication for all tables
2
create publication pub_all_tables for all tables;

This tracks changes for all tables in your database.

Advanced publication options#

Selecting specific columns

You can replicate only a subset of columns from a table:

1
-- Replicate only specific columns from the users table
2
create publication pub_users_subset
3
for table users (id, email, created_at);

This only replicates the id, email, and created_at columns from the users table.

Filtering rows with a predicate

You can filter which rows to replicate using a WHERE clause:

1
-- Only replicate active users
2
create publication pub_active_users
3
for table users where (status = 'active');
4
5
-- Only replicate recent orders
6
create publication pub_recent_orders
7
for table orders where (created_at > '2024-01-01');

Viewing publications in the Dashboard#

After creating a publication via SQL, you can view it in the Dashboard:

1. Navigate to the Database > Publications section of the Dashboard
2. You'll see all your publications listed with their tables

Step 2: Enable external replication#

Before creating an external replication pipeline, enable external replication for your project:

1. Navigate to the Database > Replication section of the Dashboard
2. Click Add destination to show the replication side panel
3. Select an external replication destination, such as BigQuery
4. Click Enable external replication

Step 3: Configure a destination#

Once external replication is enabled and you have a Postgres publication, configure a destination. The destination is where your replicated data will be stored, while the pipeline is the active Postgres replication process that continuously streams changes from your database to that destination.

Choose and configure your destination#

Follow these steps to configure your destination. Each destination has its own setup requirements and behavior. BigQuery is currently available, and we are working on new destinations.

1. Navigate to the Database > Replication section of the Dashboard

2. Click Add destination if the destination side panel isn't already open

3. Configure the general settings:

   • Destination name: A name to identify this destination
   • Publication: The publication to replicate data from (created in Step 1)
   • Destination type: Select the destination you want to use

4. Configure the destination-specific settings. See the destination guide for required credentials, permissions, and limitations:

   • BigQuery

5. Optionally expand Advanced settings to tune pipeline behavior. These settings apply to the pipeline rather than the destination:

   Setting	Default	Description
   Batch wait time	10000 milliseconds	Maximum time the pipeline waits to collect additional changes before flushing them. Lower values reduce replication latency. Higher values can improve batching efficiency.
   Table sync workers	4 workers	Number of tables copied in parallel during the initial snapshot phase. Each worker uses one replication slot, up to N + 1 total replication slots while syncing.
   Copy connections per table	2 connections	Number of parallel database connections each table copy can use during the initial sync. Increasing this can speed up large table copies, but uses more database connections.
   Invalidated slot behavior	Error	What the pipeline does when its replication slot is invalidated. Error blocks startup so you can recover manually. Recreate rebuilds the slot and starts replication from scratch.

   Leave these settings at their defaults unless you need to tune initial copy speed, latency, or recovery behavior.

6. Click Create and start to begin replication

Your replication pipeline now starts copying data from your database to your destination.

Step 4: Monitor your pipeline#

After creating a destination, the replication pipeline starts and appears in the destinations list. You can monitor the pipeline's status and performance from the Dashboard.

For comprehensive monitoring instructions including pipeline states, metrics, and logs, see the external replication monitoring guide.

Managing your pipeline#

You can manage your pipeline from the destinations list using the actions menu.

Available actions:

• Start: Begin replication for a stopped pipeline
• Stop: Pause replication (changes will queue up in the WAL)
• Restart: Stop and start the pipeline (required after publication changes)
• Edit destination: Modify destination settings like credentials or advanced options
• Delete: Remove the destination and permanently stop replication

Disabling external replication#

To turn off external replication for a project, delete all external replication pipelines first. After all pipelines are removed, open the three-dot actions menu on the Replication page and click Disable external replication.

For cleanup details, see What happens when you disable external replication?.

Adding or removing tables#

If you need to modify which tables are replicated after your replication pipeline is already running, follow these steps:

Adding tables to replication#

1. Add the table to your publication using SQL:

   1
   -- Add a single table to an existing publication
   2
   alter publication pub_users_orders add table products;
   3
   4
   -- Or add multiple tables at once
   5
   alter publication pub_users_orders add table products, categories;

2. Restart the replication pipeline using the actions menu (see Managing your pipeline) for the changes to take effect.

Removing tables from replication#

1. Remove the table from your Postgres publication using SQL:

   1
   -- Remove a single table from a publication
   2
   alter publication pub_users_orders drop table orders;
   3
   4
   -- Or remove multiple tables at once
   5
   alter publication pub_users_orders drop table orders, products;

2. Restart the replication pipeline using the actions menu (see Managing your pipeline) for the changes to take effect.

Schema change support#

Schema change support depends on the destination. BigQuery is currently the only destination with beta schema change support. See BigQuery schema change support for supported and unsupported changes.

How it works#

Once configured, external replication:

1. Captures changes from your Postgres database using Postgres publications and logical replication
2. Streams the changes through the replication pipeline
3. Loads the data to your destination

External replication automatically optimizes how changes are delivered to the destination. The replication pipeline currently performs data extraction and loading only, without transformation - your data is replicated as-is to the destination.

Troubleshooting#

If you encounter issues during setup:

• Publication not appearing: Ensure you created the Postgres publication via SQL and refresh the dashboard
• Tables not showing in publication: Verify your tables have primary keys (required for Postgres logical replication)
• Pipeline failed to start: Check the error message in the status view for specific details
• No data being replicated: Verify your Postgres publication includes the correct tables and event types

For more troubleshooting help, see the external replication FAQ.

Limitations#

External replication has the following limitations:

• Primary keys required: Tables must have primary keys (Postgres logical replication requirement)
• Custom data types: Not supported
• Schema changes: Currently in beta and limited to BigQuery
• No data transformation: Data is replicated as-is without transformation
• Data duplicates: Duplicates can occur when stopping a pipeline if your database has transactions that take longer than a few minutes to complete. See Can data duplicates occur during pipeline operations? for details

Destination-specific limitations, such as BigQuery's row size limits, are documented in each destination guide.

Next steps#

• Set up BigQuery
• Monitor external replication
• View external replication FAQ