GitHub
GitHub is where software teams host code, track issues and review pull requests. Weld syncs your GitHub organizations into your data warehouse: repositories, issues, pull requests and their reviews and commits, comments, releases, teams, members and GitHub Actions workflow runs. Use it for engineering metrics such as cycle time, review turnaround, deployment frequency and issue throughput.
Features
| Feature name | Supported | |
|---|---|---|
| Column Hashing | True | Field level |
| Blocking | True | Field level |
| Incremental | True | |
| Custom data | False | |
| History | False | |
| ReSync | True | Table level |
| Templates | False |
Data Schema
Prerequisites
- A GitHub account that can read the organizations and repositories you want to sync.
- A personal access token, fine-grained or classic (see Step 1).
- For GitHub Enterprise Server, the API URL of your server, which must be reachable from the internet.
Setup Guide
Step 1 - Create a Personal Access Token
Weld only reads from GitHub, so the token needs read access and nothing more. Create a dedicated token for Weld, ideally on a service account, so syncs don't share a rate limit with other tools (see Rate limits).
Option A - Fine-grained token (recommended)
-
In GitHub, go to Settings → Developer settings → Personal access tokens → Fine-grained tokens and click Generate new token.
-
Set Resource owner to the organization you want to sync.
-
Under Repository access, choose All repositories, or Only select repositories to limit what Weld can see.
-
Under Repository permissions, set these to Read-only:
Permission Used for Metadata Repositories (set automatically) Contents Commits, commit history and releases Issues Issues and issue comments Pull requests Pull requests, reviews, review comments and pull request commits Actions Workflow runs -
Under Organization permissions, set Members to Read-only. It is used for teams, team members and organization members.
-
Choose an expiration and click Generate token. Copy the token; GitHub shows it only once.
If your organization requires approval for fine-grained tokens, an organization owner must approve the token before it can read private repositories.
A fine-grained token belongs to one resource owner, so it can read only one organization's private repositories. To sync several organizations, create one connection per organization, or use a classic token.
Option B - Classic token
- In GitHub, go to Settings → Developer settings → Personal access tokens → Tokens (classic) and click Generate new token (classic).
- Select the scopes:
- repo, to read private repositories. If you only sync public repositories, public_repo is enough.
- read:org, to read teams and organization members.
- Choose an expiration and click Generate token, then copy it.
- If your organization uses SAML single sign-on, click Configure SSO next to the token and authorize it for the organization.
Classic tokens have no read-only option for private repositories: the repo scope also grants write access. Weld never writes to GitHub, but if your security policy requires read-only credentials, use a fine-grained token.
Step 2 - Create the Connection
-
In the Weld app, click Data Sources in the sidebar, then Add Data Source.
-
Search for GitHub, select it and choose Create new connection.
-
Fill in the form:
Field Description Personal Access Token The token from Step 1. Organizations The logins of the organizations to sync, separated by commas, for example my-org, my-other-org. Every repository the token can read in them is synced, along with their teams and members.Repositories Optional. Sync only these repositories, written as owner/nameand separated by commas, for examplemy-org/backend, my-org/frontend. Leave empty to sync every repository in the organizations.API URL Only for GitHub Enterprise Server, for example https://github.example.com/api/v3. Leave empty for github.com.Enter at least one organization or one repository.
-
Click Connect. Weld checks that the token works and can read each organization and repository you entered.
Step 3 - Select Data
- On Data to sync, tick the tables you want.
- Open a table's options to exclude or hash columns, then click Next.
Step 4 - Configure Sync
- On Configure sync, choose how often to sync.
- Check the Destination schema name, which is where the tables land in your warehouse.
- Click Start Initial Sync.
Weld starts syncing data from your GitHub organizations.
Tables
| Table | Sync mode | Description |
|---|---|---|
repositories | Full | Repositories, with their topics in repository_topics. |
issues | Incremental | Issues and pull requests (GitHub treats pull requests as issues); is_pull_request tells them apart. Labels and assignees are in issue_labels and issue_assignees. |
issue_comments | Incremental | Comments on issues and on the conversation tab of pull requests. |
pull_requests | Incremental | Pull requests, with labels, assignees and requested reviewers and teams in their own tables. |
pull_request_reviews | Incremental | Reviews submitted on pull requests. |
pull_request_review_comments | Incremental | Comments on lines of a pull request's diff. |
pull_request_commits | Incremental | The commits of each pull request. |
commits | Incremental | Commits on each repository's default branch, with their parents in commit_parents. |
releases | Full | Releases, with their assets in release_assets. |
workflow_runs | Incremental | GitHub Actions workflow runs. |
teams | Full | Teams in each organization. |
team_members | Full | Members of each team, with their role (maintainer or member). |
organization_members | Full | Members of each organization, with their role (admin or member). |
Every repository-level table has repository_id and repository_full_name, and the organization tables have organization_login, so rows from several organizations and repositories can be told apart.
How Incremental Syncs Work
- Issues, comments and pull requests are read from the time of the previous sync onwards, so each sync only reads what changed.
- Reviews and pull request commits are read only for pull requests updated since the previous sync.
- Commits are read from where the previous sync stopped on the default branch to its current head, so commits brought in by a merge are included whatever their date.
- Workflow runs created up to three days before the previous sync are read again, to pick up runs that were still in progress.
The first sync of a large organization can take a long time, mostly because of GitHub's rate limits. Weld saves its progress repository by repository, so a sync that is interrupted continues where it stopped instead of starting over.
Rate Limits
On GitHub.com, personal access token requests share a 5,000 API requests/hour limit per user; GitHub Enterprise Server may use a different configured limit. When the limit is reached, Weld waits until GitHub resets it and then continues, so syncs slow down but no data is lost. To keep syncs fast:
- Give Weld its own token, so other tools and scripts don't use up the same hourly limit.
- Use the Repositories field to sync only the repositories you need.
- Choose a sync frequency that fits the size of your organization. Each sync makes at least one request per repository for each selected table, even when nothing changed.
Limitations
commitscovers each repository's default branch. Commits on other branches appear inpull_request_commitsonce they are part of a pull request.- GitHub returns at most 250 commits for a single pull request in
pull_request_commits. - A workflow run re-run more than three days after it was created is only updated by a full re-sync of the
workflow_runstable. - When a label or assignee is removed from an issue or pull request, the old row stays in its table until a full re-sync of that table.
team_membersincludes members of child teams under their parent team, as GitHub lists them.- Pull requests don't include additions, deletions, changed files or a
mergedflag. A pull request is merged whenmerged_atis set.
Troubleshooting
"GitHub rejected the access token." The token is wrong, expired or revoked. Create a new one and update it on the connection.
"The token cannot read the organization …" or "… the repository …" Check the spelling, and that the token can reach it: for a fine-grained token, the resource owner and repository access; for a classic token, the scopes and SSO authorization.
A table is empty for some repositories. Repositories with issues turned off, without GitHub Actions, or without commits have no rows in the matching tables. They are skipped and don't fail the sync.
The teams or member tables fail or come back empty. The token needs the organization Members permission (fine-grained) or the read:org scope (classic).