AWS S3

Amazon Simple Storage Service (Amazon S3) is a cloud object storage service with industry-leading scalability, data availability, security, and performance.

Features

Supported file types

• JSON Newline
• CSV
• Parquet

Configuration options

We support many different configuration options for loading your data. If we do not support a configuration needed for your specific use case, please reach out to hello@weld.app.

🔧 Setup Guide

Step 1 - Create an IAM Policy for Weld

1. Open your AWS IAM console.
2. Go to Policies and then select Create Policy

3. Go to the JSON tab

4. Copy the following policy and paste it in the JSON tab, replacing {BUCKET_NAME} with the name of your S3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectAcl",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:GetObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::{BUCKET_NAME}",
        "arn:aws:s3:::{BUCKET_NAME}/*"
      ],
      "Condition": {
        "ForAnyValue:IpAddress": {
          "aws:SourceIp": ["35.156.133.78", "3.65.119.169", "3.64.84.139"]
        }
      }
    }
  ]
}

1. Click Next: Tags
2. Click Next: Review
3. Name the policy Weld-S3-Access
4. Click Create Policy

Step 2 - Create a User for Weld

1. Go to Users and then select Add users.

1. Name the user Weld-S3
2. Select Access key - Programmatic access as the AWS credential type

1. Click Next: Permissions
2. Click Attach existing policies directly and select the Weld-S3-Access policy
3. Click Next: Tags
4. Click Next: Review
5. Click Create user
6. Note the Access Key ID and Secret Access Key for next step.

Step 3 - Authenticate Weld

1. Input the Access Key ID from previous step in the Weld App
2. Input Secret Access Key from previous step in the Weld App
3. Input the bucket name
4. Select the region the bucket is located in.
5. Click Connect

The configuration will be finalized and you will be able to start loading data from your AWS S3 bucket.