AWS S3

Amazon Simple Storage Service (Amazon S3) is cloud object storage with industry-leading scalability, data availability, security, and performance.

Supported file types

• Avro

Configuration options

We support many different configuration options for loading your data. If we do not support a configuration needed for your specific use-case, please reach out to support@weld.app.

🔧 Setup Guide

Step 1 - Create an IAM Policy for Weld

1. Open your Amazon IAM console.
2. Go to "Policies" and then select "Create Policy":

3. In the "Specify permissions" screen, make sure you have selected "JSON":

4. Copy the following policy and paste it in the policy editor, replacing {your-bucket-name} with the name of your S3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::{your-bucket-name}"],
      "Condition": {
        "ForAnyValue:IpAddress": {
          "aws:SourceIp": ["35.156.133.78", "3.65.119.169", "3.64.84.139"]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:PutObjectAcl",
        "s3:PutObject",
        "s3:PutObjectTagging",
        "s3:GetObjectAcl",
        "s3:GetObject",
        "s3:DeleteObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging"
      ],
      "Resource": ["arn:aws:s3:::{your-bucket-name}/*"],
      "Condition": {
        "ForAnyValue:IpAddress": {
          "aws:SourceIp": ["35.156.133.78", "3.65.119.169", "3.64.84.139"]
        }
      }
    }
  ]
}

5. Click "Next: Tags".
6. Click "Next: Review".
7. Name the policy "Weld-S3-Access".
8. Click "Create Policy".

Step 2 - Create a User for Weld

1. Go to "Users" and then select "Add users":

2. Name the user "Weld-S3".
3. In the user configuration screen, select "Create access key":

4. Note the "Access Key ID" and "Secret Access Key" for next step.

Step 3 - Authenticate Weld

1. Input the "Access Key ID" from previous step in the Weld App.
2. Input "Secret Access Key" from previous step in the Weld App.
3. Input the bucket name.
4. Select the region the bucket is located in.
5. Click "Connect".