PostgreSQL on Azure Database

Azure Database for PostgreSQL offers a scalable and secure platform for hosting PostgreSQL databases in the cloud. It provides built-in high availability with automatic failover, automated backups, and intelligent performance tuning to optimize database performance. Azure Database for PostgreSQL integrates seamlessly with other Azure services, enabling developers to build comprehensive solutions. The service offers advanced security features, including data encryption at rest and in transit, and compliance with industry standards. With Azure's global presence, users can deploy PostgreSQL databases in multiple regions for improved performance and availability. PostgreSQL on Azure Database provides a reliable and efficient solution for developers and businesses seeking a managed PostgreSQL service with the flexibility and power of Microsoft Azure's cloud infrastructure.

Features

🔧 Setup Guide

Azure Database for PostgreSQL setup credentials are compatible with the generic PostgreSQL setup. By setting up an ELT sync directly with the database, we are able to achieve fast and robust ELT with deep access to the raw data.

Step 1 - Connector Configuration

From your PostgreSQL Database, access and paste the required fields:

• Name (The name you want to refer to regarding this connection in the Weld app)
• Host (IP Address of your server)
• Database (Name of the database you want to connect to)

The host can be found using PSQL commands such as:

SELECT *
FROM pg_settings
WHERE name = 'port';

DB inside private network or dynamic host/IP

Our PostgreSQL connector operates with industry standard access architecture in mind. Weld > SSH > DB

The preferred way of connecting to a PostgreSQL database source is for the user to provide a public hostname or a static IP towards an SSH server Weld can connect to.

The SSH server would be a part of the user's internal network, and/or be able to resolve the database DNS alias or IP internally. This approach is not invasive and leaves the user with full control over their own infrastructure.

Step 2 - Data To Sync

1. Select the tables you wish to include in the sync.
   You can view the schema, remove columns or hash sensitive information.
2. Update Sync mode, which can be set to:
   • Incremental Sync
   • Full Sync at Midnight

Incremental Sync

Once the initial sync is complete, Weld performs incremental updates of any new or modified data from your source database.

The hidden PostgreSQL system column xmin can be used to select only the new or changed rows since the last update.

The XMIN method has the following limitations:

• XMIN can't track row deletions because it relies on a updated_at column. There is no way to tell which rows in the destination are no longer present in the source database. It Requires a full table scan to detect and remove deleted rows.
• The xmin column is updated on a row basis. Therefore, regardless of whether you selected that column in the connector schema, if any value of a column in a row changes, the xmin value also changes which would cause the connector to sync that row.

Full Sync at Midnight

A limitation of incremental syncs are that they do not capture deleted rows. If you expect rows to be deleted often from your tables then you can set a table to run a full sync at midnight which will remove those deleted rows.

Step 3 - Configure Sync

1. Select how often you would like the data to sync.
2. Provide a unique destination table name.

Weld will take over from here and commence syncing data from your PostgreSQL database.