MySQL on Google Cloud SQL

Google Cloud SQL provides a fully managed MySQL database solution, eliminating the need for manual setup and maintenance of the infrastructure. It offers features such as automatic backups, automated software patching, and built-in high availability with options for regional replication. Google Cloud SQL allows seamless integration with other Google Cloud services and provides scalability by offering different instance sizes to meet the specific needs of applications. With Cloud SQL, developers can focus on their applications while relying on Google's infrastructure to handle database management tasks efficiently and securely. MySQL on Google Cloud SQL is widely used by developers and businesses for various applications, benefiting from the performance, reliability, and ease of use provided by Google Cloud's managed database service.

Features

🔧 Setup Guide

Google Cloud SQL database's setup credentials are compatible with the generic MySQL setup. By setting up an ELT sync directly with the database, we are able to achieve fast and robust ELT with deep access to the raw data.

To find out how to define the connector configuration, follow the MySQL connector's setup steps:

Creating a privileged user

The preferred way of creating a new user is directly through a SQL statement, where you can additionally tailor the access levels:

CREATE USER 'my_user'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'my_user'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Optionally, you can also use the GCP UI:

• Go to your database's page, under Connections → Users
• Add credentials for the new user account.

Setup Instructions

Step 1 - Get credentials to setup the connection

From your MySQL Database, access and paste the required fields:

• Name (The name you want to refer to regarding this connection in the Weld app)
• Host (IP Address of your server)
• Database (Name of the database you want to connect to)

The host can be found running the following SQL commands in your MySQL database:

select @@hostname;
show variables where Variable_name like '%host%';

Furthermore, add the port, name and your Weld username and password if desired. You can also connect through an SSH Tunnel.

Step 2 - Data To Sync

1. Select the tables you wish to include in the sync.
   You can view the schema, remove columns or hash sensitive information.
2. Update Sync mode:
   By default the MySQL connector is set to run always full syncs. To optimize the sync time and reduce processing overhead on your MySQL server we recommend for you to set up the syncs to run incrementally.

We currently support both Merge and Append mode for incremental syncs.

Merge

To have your table running incrementally using the merge configuration you need a table primary key and a cursor timestamp (updated_at is preferred).

When a sync is run, Weld will select only the new or changed rows since the last update.

Append

If a row updated_at timestamp is not available on the table then another option is to run an incremental sync using append mode. Append mode uses cursor to keep track of how far the sync got on the last run. It will use that cursor to append new entries at the end of the table on the next run.

Append is not widely used as it does not capture updates in the previous rows.

Full Sync at Midnight

A limitation of both Merge and Append modes is that they do not capture deleted rows. If you expect rows to be deleted often from your tables then you can set a table to run a full sync at midnight which will remove those deleted rows.

Step 3 - Configure Sync

1. Select how often you would like the data to sync.
2. Provide a unique destination table name.

Weld will take over from here and commence syncing data from your MySQL database.