AWS S3
Amazon Simple Storage Service (Amazon S3) is cloud object storage with industry-leading scalability, data availability, security, and performance.
Supported file types
- Avro
Configuration options
We support many different configuration options for loading your data. If we do not support a configuration needed for your specific use-case, please reach out to support@weld.app.
π§ Setup Guide
Prerequisites
An empty S3 bucket where we will send files to.
Step 1 - Create an IAM Policy for Weld
- Open your Amazon IAM console.
- Go to "Policies" and then select "Create Policy"
- Go to the "JSON tab"
- Copy the following policy and it in the JSON tab, replacing {BUCKET_NAME} with the name of your S3 bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ReplicateObject",
"s3:PutObjectAcl"
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:DeleteObjectTagging",
],
"Resource": [
"arn:aws:s3:::{BUCKET_NAME}",
"arn:aws:s3:::{BUCKET_NAME}/*"
],
"Condition": {
"ForAnyValue:IpAddress": {
"aws:SourceIp": ["35.156.133.78", "3.65.119.169", "3.64.84.139"]
}
}
}
]
}
- Click "Next: Tags"
- Click "Next: Review"
- Name the policy "Weld-S3-Access"
- Click "Create Policy"
Step 2 - Create a User for Weld
- Go to "Users" and then select "Add users"
- Name the user "Weld-S3"
- Select "Acces key - Programmatic access" as the AWS credential type
- Click "Next: Permissions"
- Click "Attach existing policies directly" and select the "Weld-S3-Access" policy
- Click "Next: Tags"
- Click "Next: Review"
- Click "Create user"
- Note the "Access Key ID" and "Secret Access Key" for next step
Step 3 - Authenticate Weld
- Input the "Access Key ID" from previous step in the Weld App
- Input "Secret Access Key" from previous step in the Weld App
- Input the bucket name
- Select the region the bucket is located in.
- Click "Connect"