DoubleCloud
Overview
DoubleCloud helps businesses harness their data by providing a set of managed open-source solutions. They offer managed services for data integration, orchestration, storage, analytics, and visualization. DoubleCloud is an engineer-friendly, easy-to-use platform that lets you build data analytics infrastructure in one day.
With DoubleCloud, you can replicate your data in real time from Supabase to ClickHouse — a database designed for analytics. This will enable you to run analytical queries faster, use less storage, and save costs while still benefiting from using an OLTP database.
Documentation
DoubleCloud is a managed data platform that helps engineering teams build data infrastructure with zero-maintenance open-source technologies, such as ClickHouse.
ClickHouse is an open-source column-oriented database that's able to query billions of rows in milliseconds. It's designed to efficiently handle analytical queries, which makes it the go-to solution for data warehousing and analytics applications. Thanks to the columnar storage format, data can be compressed and retrieved more efficiently, allowing some analytical queries to execute 100 times faster compared to traditional databases like Postgres.
This tutorial guides you through creating a Managed ClickHouse cluster and transferring data to it with DoubleCloud Transfer — a real-time data replication tool. Transfer natively supports ClickHouse data types, data mutations, automated migrations (adding columns), as well as emulating insertions and deletions.
Before you start
-
If you haven't already, create a DoubleCloud account.
If you're a new DoubleCloud user, this tutorial won't incur you any costs — you can use the trial period credits to test the platform, including creating fully operational clusters.
-
If you don't have a Supabase instance with data that you want to transfer, load sample data to your project. To do that:
-
Open your project in Supabase and click SQL editor in the left menu.
-
Click Quickstarts → Countries.
-
Click Run.
When the data is loaded into the
public.countries
table, you'll see theSuccess. No rows returned
message.
-
Step 1. (Optional) Enable real-time replication
You need to enable replication if you want to transfer a snapshot of the existing data and then transfer all eventual incremental updates. If don't want to transfer updates, you can skip this step.
To enable real-time replication:
-
Open your project in Supabase and click Table editor in the left menu.
-
Select your table and click the realtime button at the top right.
-
In the dialog, confirm enabling realtime for the table.
Step 2. Enable access to Supabase and get connection details
By default, databases in your Supabase project have no access restrictions and can be accessed by any IP address.
-
If you have limited access to the database, add Transfer's IP addresses to the allowlist. To do that:
-
Open your project in Supabase, click Project settings in the left menu and select Database
-
Scroll down to the Network restrictions section and click Add restriction.
-
Add Transfer's IP addresses:
_10# IPv6_102a05:d014:e78:3500::/56_10# IPv4_103.77.1.232_103.74.181.206_103.78.156.2_103.77.29.32_103.125.212.122
-
-
Open your project in Supabase, click Project settings in the left menu and select Database
-
Under Database settings, copy the URI connection string. You will need this string when configuring the source endpoint later in this tutorial.
Step 3. Create a Managed ClickHouse cluster
-
In the DoubleCloud console, go to the Clusters page.
-
Click Create cluster at the top right and select ClickHouse.
You can proceed with the default settings on this page to get a fully functional ClickHouse cluster that you can use for testing and development.
When creating a production cluster, refer to the Create a Managed ClickHouse cluster guide for a list of ways to create and configure a cluster.
-
Under Basic settings, enter a cluster name, such as
clickhouse-dev
. -
Click Submit.
Creating a cluster takes around five minutes depending on the provider, region, and settings.
Step 3. Create a ClickHouse database
-
After the cluster status changes from Creating to Alive, select it in the cluster list.
-
Click WebSQL at the top right.
-
In WebSQL, click on any database in the connection manager on the left to open the query editor.
-
Create a new database named
listings
using the following command in the query editor:_10CREATE DATABASE IF NOT EXISTS countries ON CLUSTER default; -
Check that the database has been created:
_10SHOW DATABASES_10┌─name───────────────┐_10│ INFORMATION_SCHEMA │_10│ _system │_10│ default │_10│ countries │ // your database_10│ information_schema │_10│ system │_10└────────────────────┘
Step 4. Create a source PostgreSQL endpoint
Because Supabase is essentially a Postgres database, you can connect to it by configuring a source PostgreSQL endpoint.
To create a source endpoint:
-
Go to the Transfer page in the DoubleCloud console and click Create → Source endpoint at the top right.
-
In Source type, select PostgreSQL.
-
Enter an endpoint name, such as
supabase-source-dev
. -
Under Endpoint parameters, fill out connection details for your Supabase database. You can get the connection details from the connection string that you copied earlier. This string has the following format:
_10postgres://<username>:<password>@<hostname>:<port>/<database> -
Under Collection filter, add
countries
in Included collections. -
Click Submit.
Step 5. Create a target ClickHouse endpoint
To create a target endpoint:
-
On the Transfer page, click Create → Target endpoint.
-
In Target type, select ClickHouse.
-
Enter an endpoint name, such as
clickhouse-target-dev
. -
In Connection type, select Managed cluster and select the cluster that you created from the dropdown.
You can leave the default values in other fields.
-
Click Submit.
Step 6. Create a transfer
Now that your endpoints are ready, you can create a transfer.
-
On the Transfer page, click Create → Transfer at the top right.
-
Under Endpoints, select the endpoints you just created — supabase-source-dev and clickhouse-target-dev as the source and target respectively.
-
Under Basic settings, enter a transfer name, such as
supabase-clickhouse-dev
. -
In Transfer type, select Snapshot and replication.
This transfer type transfers the current state of the source database, and then keeps all the added data in sync.
-
Leave the default preselected options in Snapshot settings and Runtime environment
-
In List of objects for transfer, click + Object and enter the name of the collection you want to transfer —
public.countries
. -
Click Submit.
Step 7. Activate the transfer and query transferred data
Now that you transfer is fully configured, you can use it to replicate data from Supabase to ClickHouse.
-
On the transfer page, click Activate at the top right.
After the transfer is activated, it fetches data from the source database and uploads it to the target database.
The initial activation may take a few minutes. When some data is successfully replicated, you can see green bars in the Status history section.
-
Go to the Clusters page in the DoubleCloud console.
-
Select the cluster you created earlier.
-
Click WebSQL at the top right.
-
In the Connection manager on the left, select the countries database.
-
In the SQL editor, run a
SELECT
query:_10SELECT * FROM public_countries LIMIT 100; -
The query output is displayed in the Results section.
What's next
-
Create a Managed ClickHouse cluster on DoubleCloud for your production data
-
Learn how to use DoubleCloud Transfer to replicate, migrate, and transform data
For more implementation guides, code examples, and support, visit DoubleCloud documentation or join the Slack community.
Details
Third-party integrations and docs are managed by Supabase partners.