DynamoDB

DynamoDB is a managed NoSQL database service provided by Amazon Web Services (AWS). Designed for high availability, consistent performance, and seamless scalability, DynamoDB allows users to create database tables that can store and retrieve any amount of data and serve any level of request traffic. With its fully managed infrastructure, developers do not need to worry about database operations like hardware provisioning, patching, or replication. It supports both document and key-value data models, making it versatile for various application needs, from gaming and mobile apps to large-scale web applications.

Features

🔧 Setup Guide

Step 1 - Create IAM role

1. Open the Create new AWS IAM role page.
2. When you setup the role you need to attach it to Weld AWS account with id: 887019001183
3. Then you need to add the external id as seen on the connector setup page. Then click Next.

Role setup

Create a new role inside of your AWS account.

Open the Create new AWS IAM policy page.

When you setup the role you need to attach it to Weld AWS account with id: 887019001183

This is added for extra layer of protection.

Go to the JSON tab.

The permission you need to attach to the role looks like this:

{
    "Statement": [
        {
            "Action": [
                "dynamodb:DescribeStream",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:GetRecords",
                "dynamodb:GetShardIterator",
                "dynamodb:ListStreams",
                "dynamodb:ListTables",
                "dynamodb:Scan"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Turn on dynamodb streams

You navigate to https://console.aws.amazon.com/dynamodbv2/home and choose Update Settings under tables.
Select the table(s) you want to sync and go to Exports and streams there you find DynamoDB stream details
You need to activate the stream with View type: New and old images
And should look like this when setup correctly:

Step 2 - Data To Sync

1. Select the tables you wish to include in the sync.
   You can view the schema, remove columns or hash sensitive information.
2. Update Sync mode:
   By default the MySQL connector is set to run always full syncs. To optimize the sync time and reduce processing overhead on your MySQL server we recommend for you to set up the syncs to run incrementally.

We currently support both Merge and Append mode for incremental syncs.

Merge

To have your table running incrementally using the merge configuration you need a table primary key and a cursor timestamp ( updated_at is preferred )

When a sync is run, Weld will select only the new or changed rows since the last update.

Append

If a row updated_at timestamp is not available on the table then another option is to run an incremental sync using append mode. Append mode uses cursor to keep track of how far the sync got on the last run. It will use that cursor to append new entries at the end of the table on the next run.

Append is not widely used as it does not capture updates in the previous rows

Full Sync at Midnight

A limitation of both Merge and Append modes are that they do not capture deleted rows. If you expect rows to be deleted often from your tables then you can set a table to run a full sync at midnight which will remove those deleted rows.

Step 3 - Configure Sync

1. Select how often you would like the data to sync.
2. Provide a unique destination table name.

Weld will take over from here and commence syncing data from your DynamoDB database.