AWS S3

Amazon Simple Storage Service (Amazon S3) is cloud object storage with industry-leading scalability, data availability, security, and performance.

Supported file types

  • Avro

Configuration options

We support many different configuration options for loading your data. If we do not support a configuration needed for your specific use-case, please reach out to support@weld.app.

πŸ”§ Setup Guide

Step 1 - Create an IAM Policy for Weld

  1. Open your Amazon IAM console.
  2. Go to "Policies" and then select "Create Policy"
Amazon IAM Policies
  1. Go to the "JSON tab"
Amazon JSON tab
  1. Copy the following policy and it in the JSON tab, replacing {BUCKET_NAME} with the name of your S3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:PutObjectAcl"
        "s3:PutObject",
        "s3:PutObjectTagging",
        "s3:GetObjectAcl",
        "s3:GetObject",
        "s3:DeleteObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectTagging",
      ],
      "Resource": [
        "arn:aws:s3:::{BUCKET_NAME}",
        "arn:aws:s3:::{BUCKET_NAME}/*"
      ],
      "Condition": {
        "ForAnyValue:IpAddress": {
          "aws:SourceIp": ["35.156.133.78", "3.65.119.169", "3.64.84.139"]
        }
      }
    }
  ]
}

  1. Click "Next: Tags"
  2. Click "Next: Review"
  3. Name the policy "Weld-S3-Access"
  4. Click "Create Policy"

Step 2 - Create a User for Weld

  1. Go to "Users" and then select "Add users"
Amazon Add users
  1. Name the user "Weld-S3"
  2. Select "Acces key - Programmatic access" as the AWS credential type
Amazon user setup
  1. Click "Next: Permissions"
  2. Click "Attach existing policies directly" and select the "Weld-S3-Access" policy
  3. Click "Next: Tags"
  4. Click "Next: Review"
  5. Click "Create user"
  6. Note the "Access Key ID" and "Secret Access Key" for next step

Step 3 - Authenticate Weld

  1. Input the "Access Key ID" from previous step in the Weld App
  2. Input "Secret Access Key" from previous step in the Weld App
  3. Input the bucket name
  4. Select the region the bucket is located in.
  5. Click "Connect"

Was this page helpful?