FTP

FTP provides a straightforward method for users to upload, download, and manage files on remote servers. FTP operates on a client-server model, where the client initiates a connection to the server and performs various file operations. It offers authentication mechanisms for secure access control, ensuring that only authorized users can access the files. FTP supports both binary and text file transfers, allowing for the transfer of a wide range of file types. While FTP is an older protocol and lacks some of the advanced security features of newer protocols, it remains widely used for its simplicity and compatibility with various operating systems and applications.

πŸ”§ Setup Guide

Step 1 - Begin Configuration

  1. After selecting Data Sources in the side menu, click + New on the top right.
  2. Find the FTP connector from the list and click on it.
  3. In the connector setup form, enter the connection name of your choice.
  4. Fill in the required fields:
  • Host - the IP address or domain name of the FTP server.
  • User - the username used to authenticate with the FTP server.
  • Password - the password used to authenticate with the FTP server.
  • Port - the port number used for the FTP connection.

Step 2 - Connection Settings

Fill in the following fields:

  • Table name - the name of the table you want to create in the destination schema.
  • File type - the file type of the files you want to load (currently only supports CSV).
  • File Regex pattern - a regular expression pattern to match the files you want to load.
    • It will only select files in the root folder.
    • Make sure all the files have the same schema (same columns and same data types).
    • An example regex that finds all CSV files is .*csv

Tip: You can switch to a different folder by changing the root folder for the FTP User in the server.

Optionally, you can also fill in the following:

  • Delimiter - the delimiter used in the files.

Step 3 - Data to sync

  1. Select the data you want to sync from the FTP server to the destination schema.
  2. You can view the schema, remove columns, and setup the type of sync:
  • Full sync - sync all data from the source to the destination.
  • Incremental sync - sync only new or updated data from the source to the destination. This requires setup of Primary Key column(s) and Cursor.

Weld fields:

We automatically add the following fields to the schema:

  • _file: the file name
  • _modified: the timestamp when the file was last modified
  • _line: the line number of the record in the CSV file

By default you can use the _file and _line field as the primary keys for incremental sync, as these will be unique for each record, or you can choose your own primary key.
The _modified field can be used as the cursor for incremental sync, as it will track the last modified time of the file. Otherwise, you can use any timestamp or incremental field in your data.

Step 4 - Configure sync

  1. Select how often you would like the data to sync.
  2. Provide a unique destination schema name.

Weld will take over from here and commence syncing data from your FTP server.

Was this page helpful?