Adding pagination to your custom connector

✅ What you’ll learn in this module:

• Why pagination matters
• How Weld uses the state field to track sync progress
• How to return different pages of data with each call
• How to test paginated responses locally

🔁 Why paginate?

APIs often return data in pages to avoid overloading the system. Weld expects your connector to do the same — especially for tables with many rows.

You handle pagination using the state object Weld sends and expects back.

🧱 Example

Let’s say you have 1000 records, but you only return 2 at a time.

• On the first call, Weld sends an empty state: { "state": {} } → You return page 1 of results, set "state": { "page": 2 }, and hasMore: true
• On the next call, Weld sends: { "state": { "page": 2 } } → You return page 2 of results, update the state again

When you reach the last page, set hasMore: false.

🛠 Add pagination logic to your connector

Update your get_data() function like this:

1def get_data(request_data):
2    table = request_data.get("name")
3    state = request_data.get("state", {})
4    current_page = state.get("page", 1)
5
6    # Some example data
7    all_data = [
8        {"id": "123", "title": "Form A", "created_at": "2024-12-01T10:00:00Z"},
9        {"id": "124", "title": "Form B", "created_at": "2024-12-02T11:00:00Z"},
10        {"id": "125", "title": "Form C", "created_at": "2024-12-03T09:00:00Z"},
11        {"id": "126", "title": "Form D", "created_at": "2024-12-04T08:00:00Z"}
12    ]
13
14    page_size = 2
15    start = (current_page - 1) * page_size
16    end = current_page * page_size
17    page_data = all_data[start:end]
18
19    has_more = end < len(all_data)
20    next_state = {"page": current_page + 1} if has_more else {}
21
22    return {
23        "insert": page_data,
24        "state": next_state,
25        "hasMore": has_more
26    }
27

🧪 Test in Postman

1. First request body:

1{
2  "name": "forms",
3  "state": {}
4}

→ Should return rows 1–2, hasMore: true, state: { "page": 2 }

2. Second request body:

1{
2  "name": "forms",
3  "state": {
4    "page": 2
5  }
6}

→ Should return rows 3–4, hasMore: false, state: {}

✅ You now have working pagination logic!