dbt Orchestration overview

With the dbt connector, you can easily orchestrate your dbt transformations within Weld. This integration allows you to trigger dbt jobs directly from Weld, enabling a seamless workflow for your data transformations. This integration combines the power of dbt's modular, version-controlled transformations with Weld's ingestion and reverse ETL capabilities, ensuring your data is always consistent and synced across systems.

Key features

  • Fully managed by dbt Labs (formerly known as dbt Cloud)
  • Modular SQL models & Jinja macros
  • Git-based version controlled
  • Weld powers ingestion & reverse ETL
  • Trigger dbt jobs directly from Weld

dbt

Prerequisites

  1. A Weld account connected to your data warehouse.
  2. Create an Orchestration Workflow in Weld. At the most basic this could be a workflow with a single Data Source (ELT) sync running on the orchestration schedule. This will be the sync job which will later be triggering the webhook to dbt.
  3. You will need to have a dbt account with the following:
  • API credentials and tokens (for instructions on how to create API and token check dbt documentation under 'Connectors')
  • You will also need at least one or more tranformation projects ready for orchestration
  1. Lastly an access is required to your dbt repository (e.g. GitHub, GitLab) containing your project files.

πŸ”§ Setup Guide - Set up the dbt integration in Weld

Step 1 - Create a Weld Account

If you haven't already, create a Weld Account and log into your workspace.

Step 2: Find your API server region

Depending on your API server URL, the region is either us, eu or au. You can find this information in your dbt Cloud account settings or by checking the URL:

  • us: https://cloud.getdbt.com
  • eu: https://emea.dbt.com
  • au: https://au.dbt.com

Alternatively, if you have custom dbt URL, please provide it here. Leave empty if you are using the default URL. For example: if your URL is https://ti014.us1.dbt.com, then input: ti014

Step 3: Create token

You can find out more about service account tokens in the dbt Cloud documentation.

  1. Go to your dbt Cloud account settings.

  2. Click on Account settings.

  3. Click on Service tokens and select + Create service token.

  4. After clicking on + Add, enter the token name and select the following permissions:

    • Member for accounts on the Team plan
    • Developer for accounts on the Enterprise plan

  5. In the Project dropdown, choose All projects.

  6. Click on Save to generate the token and make a note of it.

Step 4: Set up the dbt integration in Weld

  1. Go to the Settings page in Weld and click on Transform.

  2. Select dbt from the list and click Connect dbt.

  3. For setting up dbt enter the following details:

    • Name: A name for the integration.
    • Token: The token you generated in Step 4.
    • dbt region: The region of your dbt API server.
    • Account prefix: For custom dbt URLs. Leave empty if you are using the default URL.

  4. Click on Connect to complete the setup.

Step 5: Sync your dbt jobs

Once you have set up the dbt integration in Weld, you can sync your dbt jobs to start transforming and loading your data. To do so, follow these steps:

  1. Go to the Orchestration page in Weld.
  2. Click on + Add Orchestration
  • Orchestration Name: A name for the orchestration.
  • Frequency: The frequency at which the orchestration should run.
  1. In the Webhooks section, select Create webhook:
  • Name: A name for the webhook.
  • External system: Select dbt.
  • Select dbt job: Choose the dbt job you want to sync.
  • Select job(s) to run after: Choose the job(s) to run after the selected job. The webhook will trigger after these jobs complete successfully.
  1. Click on Save to create the webhook.

That's it! Your dbt job will now be synced with Weld, and you can monitor its progress on the Orchestration page.

Benefits

  • Once connected there is no more manual coordination between Weld and dbt.
  • You will have automated, end-to-end workflows from ingestion to transformation to reverse ETL.
  • Logs and jobs statuses remain available in both Weld and dbt for full visibility.

Was this page helpful?