✅ What you’ll learn in this module:

  • How Weld handles errors, deduplication, and retries
  • How to implement rate limiting and pagination in real-world APIs
  • How to version and manage your connector setup using Git

🔁 Deduplication (optional)

Some APIs may send duplicate records (especially when paging or syncing often).

Weld includes an optional deduplication feature when you create a custom connector sync. Enable this if:

  • Your source API doesn’t guarantee unique results
  • You’re syncing based on time ranges (like updated_at) and might fetch overlapping windows

To help Weld deduplicate, make sure:

  • You define a primary key in your /schema response
  • That primary key is stable and truly unique

🚨 Error handling

If your connector returns an error (like a timeout or invalid auth), Weld will:

  • Retry the sync up to 3 times
  • Alert you in the Weld UI
  • Automatically stop the sync if repeated failures occur

You can see logs and failures in the Sync History section.


⏱ Rate limiting

APIs often restrict how many calls you can make per hour or minute. If you exceed this, you might get a 429 Too Many Requests error.

Best practices:

  • Track rate limit headers from the API (e.g. X-RateLimit-Remaining)
  • Implement delays using time.sleep() in Python
  • If needed, pause your sync when nearing limits

Example:

1if ratelimit_remaining < 5:
2    time.sleep(60)  # wait a minute before continuing
3


🧠 Sync state strategies

Weld sends a state object with every request — you decide how to use it.

Common patterns:

  • Use updated_at timestamps to track what’s new
  • Use next_page to handle offset- or cursor-based pagination
  • Track complex flows by storing multiple state values (e.g. { "page": 3, "last_sync": "2024-10-01" })

🛡 Version control with Git (optional)

If you’re managing multiple connectors or updating models regularly, Weld supports Git-based workflows. This lets you:

  • Track changes in your models
  • Use branches for staging vs. production
  • Roll back safely

You can configure Git sync under Settings → Git Integration in Weld.


✅ You’ve completed the course!

By now, you’ve learned how to:

  • Build and host your own connector
  • Handle pagination, state, and authentication
  • Connect it to Weld
  • Transform and use the data in your warehouse

Your connector is now fully production-ready 🎉