DuckLake destination

Replicate Postgres tables to DuckLake for current-state lakehouse queries. Prepare resources, configure the destination, then query replicated data.

Source table requirements#

Insert-only tables don't require a primary key or replica identity. Updates and deletes require a published Postgres row identity. See supported replica identities.

Prepare DuckLake resources #

DuckLake stores metadata in a Postgres catalog and data in object storage. Pipelines offers two ways to set them up:

• Use this Supabase project: Pipelines uses this project's Postgres database as the catalog and a Storage bucket in the same project as object storage. It also creates the connection credentials.
• Enter connection details: You provide the catalog URL and the storage credentials.

To read the replicated data, you also need a compatible query engine.

You can query replicated tables, but treat them and their underlying catalog and object-storage state as read-only. Writes outside Pipelines can conflict with replication and background maintenance.

Configure DuckLake as a destination #

Follow Set up Pipelines. In Add pipeline, select DuckLake, enter a Pipeline name, choose a Publication, and set Initial sync. Then choose how to configure the catalog and storage.

Use this Supabase project #

Use the current Supabase project's Postgres database for the catalog and a Storage bucket in the same project. Pipelines creates the connection credentials when you create the destination.

Before you begin:

• Choose a Files bucket for DuckLake data. You can create a private bucket from the destination form.
• Choose an unused metadata schema name for this DuckLake. Pipelines creates the schema and its tables. Use only letters, numbers, and underscores. If the project already uses the default ducklake schema for Warehouse, enter a different name.

To configure the destination:

1. Select Use this Supabase project under Configuration method.
2. Enter the Metadata schema.
3. Choose the Bucket. Select New bucket at the bottom of the bucket list if you haven't created one yet.
4. Optionally adjust Pool size under Advanced settings. It allows 1 to 6 concurrent DuckDB connections; the default is 4.
5. Click Start pipeline. Review any validation warnings, then confirm the estimated cost with Create and start pipeline.

Credential-provisioning warnings are expected before creation: catalog and Storage credentials are provisioned when you save the destination. Review the metadata schema and bucket before proceeding.

Enter connection details #

Use this mode to connect an existing Postgres database and S3-compatible object storage. You provide their connection details and credentials.

Prepare the following resources:

1. A Postgres database reachable from managed Pipelines. Create a dedicated user with permission to create the DuckLake metadata schema and its tables.
2. An S3-compatible bucket and a dedicated prefix for this DuckLake.
3. Object-storage credentials that can list, read, write, and delete objects under that prefix. Delete access is required for managed file cleanup.

Choose an unused metadata schema name and a new data prefix for each destination. Pipelines creates the schema and its tables. Initializing a DuckLake catalog in that schema beforehand triggers a validation warning; reusing its schema or prefix can mix metadata or files from different DuckLakes.

Select Enter connection details under Configuration method, then configure these fields:

• Catalog URL: A postgres:// or postgresql:// URL for your existing database, including credentials. If your provider requires TLS, append ?sslmode=require after the database name, or &sslmode=require if the URL already has query parameters. Keep the database name from your provider's URL; it doesn't need to be ducklake_catalog.
• Data path: An s3://<bucket>/<prefix> URL
• S3 access key ID and S3 secret access key: A credential pair for the data path
• S3 endpoint: A publicly reachable provider endpoint without http:// or https://
• S3 region: The storage provider's region
• S3 URL style: Path style if the bucket is in the URL path, or virtual-host style if the bucket is in the hostname
• Use SSL: Choose On for HTTPS, or Off if your provider requires HTTP
• Metadata schema: A unique Postgres schema for DuckLake metadata, using only letters, numbers, and underscores

Optionally adjust Pool size under Advanced settings. It allows 1 to 6 concurrent DuckDB connections; the default is 4. Click Start pipeline, review the validation results, then confirm the estimated cost with Create and start pipeline.

The catalog URL and storage credentials are stored as secrets and aren't returned after creation. When editing the destination, leave a secret field empty to keep its stored value, or enter a new value to replace it.

How replication works#

Pipelines creates current-state tables, copies rows according to the initial sync selection, then applies published changes and supported schema changes.

Source schema and table names are preserved, while ASCII uppercase letters in column names are converted to lowercase. DuckDB compares schema, table, and column identifiers without ASCII case distinctions, so don't publish names that differ only by ASCII case. Use distinct lowercase column names. Source column names matching the generated supabase_etl_ducklake_dropped_<ordinal>_<hash> shape are reserved for schema-change recovery.

A source TRUNCATE truncates the DuckLake table. A table restart drops and recreates it. Removing a table from the publication leaves its destination data in place.

Query replicated data #

Connect DuckDB with its ducklake extension, or another compatible engine, to the same catalog and storage path. Query through the catalog; reading raw Parquet files can miss inlined changes, delete files, and the current snapshot.

If you use this Supabase project, create separate read credentials for its Postgres database and Storage. The writer credentials generated for Pipelines aren't exposed. If you enter connection details, use separate read-only credentials when your catalog and storage provider support them.

After attaching the catalog under an alias such as my_ducklake, source schemas and tables are available as qualified DuckLake tables:

1
select *
2
from my_ducklake.public.orders;

See the DuckDB connection guide for the current ducklake extension and ATTACH syntax. If your query client can't read a Supabase-backed catalog during public alpha, contact support.

Replica identity#

Replica-identity changes affect only new WAL. If retained updates or deletes still fail after a pipeline restart, restart replication for the affected table.

Type mapping#

Pipelines creates DuckLake columns with these mappings:

DuckDB decimals support precision from 1 to 38 and a scale between 0 and the precision. Postgres numeric values declared outside that range, unconstrained numeric, and numeric types with unsupported modifiers are stored as varchar to preserve their serialized value.

Numeric arrays use varchar[], even when their declared precision and scale would fit a scalar DuckLake decimal.

Schema change support#

Pipelines supports:

• Adding, renaming, or dropping columns
• Dropping NOT NULL from an existing column
• Adding, changing, or removing supported column defaults
• Adding or removing published columns on tracked tables

New columns are created as nullable so existing destination rows remain valid. Adding NOT NULL keeps an existing destination column nullable. Supported defaults are stored in DuckLake metadata; other defaults are skipped. Postgres still supplies the source values through replication.

Previously excluded columns are added without a default, leaving historical rows NULL. Removing a published column drops its destination values; adding it again does not restore them.

For type changes, unsupported changes, and interrupted schema changes, see the shared schema-change behavior and recovery.

Troubleshooting#

Use pipeline monitoring to inspect errors. For unresolved failures, contact support with the pipeline ID and error details.

Additional resources#

• DuckLake documentation
• DuckLake architecture
• Monitor pipeline status