GitHub

GitHub is where software teams host code, track issues and review pull requests. Weld syncs your GitHub organizations into your data warehouse: repositories, issues, pull requests and their reviews and commits, comments, releases, teams, members and GitHub Actions workflow runs. Use it for engineering metrics such as cycle time, review turnaround, deployment frequency and issue throughput.

Features

Feature nameSupported
Column HashingTrueField level
BlockingTrueField level
IncrementalTrue
Custom dataFalse
HistoryFalse
ReSyncTrueTable level
TemplatesFalse

Data Schema


Setup Guide

Step 1 - Create a Personal Access Token

Weld only reads from GitHub, so the token needs read access and nothing more. Create a dedicated token for Weld, ideally on a service account, so syncs don't share a rate limit with other tools (see Rate limits).

Option A - Fine-grained token (recommended)

  1. In GitHub, go to Settings → Developer settings → Personal access tokens → Fine-grained tokens and click Generate new token.

  2. Set Resource owner to the organization you want to sync.

  3. Under Repository access, choose All repositories, or Only select repositories to limit what Weld can see.

  4. Under Repository permissions, set these to Read-only:

    PermissionUsed for
    MetadataRepositories (set automatically)
    ContentsCommits, commit history and releases
    IssuesIssues and issue comments
    Pull requestsPull requests, reviews, review comments and pull request commits
    ActionsWorkflow runs
  5. Under Organization permissions, set Members to Read-only. It is used for teams, team members and organization members.

  6. Choose an expiration and click Generate token. Copy the token; GitHub shows it only once.

If your organization requires approval for fine-grained tokens, an organization owner must approve the token before it can read private repositories.

Option B - Classic token

  1. In GitHub, go to Settings → Developer settings → Personal access tokens → Tokens (classic) and click Generate new token (classic).
  2. Select the scopes:
    • repo, to read private repositories. If you only sync public repositories, public_repo is enough.
    • read:org, to read teams and organization members.
  3. Choose an expiration and click Generate token, then copy it.
  4. If your organization uses SAML single sign-on, click Configure SSO next to the token and authorize it for the organization.

Classic tokens have no read-only option for private repositories: the repo scope also grants write access. Weld never writes to GitHub, but if your security policy requires read-only credentials, use a fine-grained token.

Step 2 - Create the Connection

  1. In the Weld app, click Data Sources in the sidebar, then Add Data Source.

  2. Search for GitHub, select it and choose Create new connection.

  3. Fill in the form:

    FieldDescription
    Personal Access TokenThe token from Step 1.
    OrganizationsThe logins of the organizations to sync, separated by commas, for example my-org, my-other-org. Every repository the token can read in them is synced, along with their teams and members.
    RepositoriesOptional. Sync only these repositories, written as owner/name and separated by commas, for example my-org/backend, my-org/frontend. Leave empty to sync every repository in the organizations.
    API URLOnly for GitHub Enterprise Server, for example https://github.example.com/api/v3. Leave empty for github.com.

    Enter at least one organization or one repository.

  4. Click Connect. Weld checks that the token works and can read each organization and repository you entered.

Step 3 - Select Data

  1. On Data to sync, tick the tables you want.
  2. Open a table's options to exclude or hash columns, then click Next.

Step 4 - Configure Sync

  1. On Configure sync, choose how often to sync.
  2. Check the Destination schema name, which is where the tables land in your warehouse.
  3. Click Start Initial Sync.

Weld starts syncing data from your GitHub organizations.


Tables

TableSync modeDescription
repositoriesFullRepositories, with their topics in repository_topics.
issuesIncrementalIssues and pull requests (GitHub treats pull requests as issues); is_pull_request tells them apart. Labels and assignees are in issue_labels and issue_assignees.
issue_commentsIncrementalComments on issues and on the conversation tab of pull requests.
pull_requestsIncrementalPull requests, with labels, assignees and requested reviewers and teams in their own tables.
pull_request_reviewsIncrementalReviews submitted on pull requests.
pull_request_review_commentsIncrementalComments on lines of a pull request's diff.
pull_request_commitsIncrementalThe commits of each pull request.
commitsIncrementalCommits on each repository's default branch, with their parents in commit_parents.
releasesFullReleases, with their assets in release_assets.
workflow_runsIncrementalGitHub Actions workflow runs.
teamsFullTeams in each organization.
team_membersFullMembers of each team, with their role (maintainer or member).
organization_membersFullMembers of each organization, with their role (admin or member).

Every repository-level table has repository_id and repository_full_name, and the organization tables have organization_login, so rows from several organizations and repositories can be told apart.

How Incremental Syncs Work

  • Issues, comments and pull requests are read from the time of the previous sync onwards, so each sync only reads what changed.
  • Reviews and pull request commits are read only for pull requests updated since the previous sync.
  • Commits are read from where the previous sync stopped on the default branch to its current head, so commits brought in by a merge are included whatever their date.
  • Workflow runs created up to three days before the previous sync are read again, to pick up runs that were still in progress.

The first sync of a large organization can take a long time, mostly because of GitHub's rate limits. Weld saves its progress repository by repository, so a sync that is interrupted continues where it stopped instead of starting over.

Rate Limits

On GitHub.com, personal access token requests share a 5,000 API requests/hour limit per user; GitHub Enterprise Server may use a different configured limit. When the limit is reached, Weld waits until GitHub resets it and then continues, so syncs slow down but no data is lost. To keep syncs fast:

  • Give Weld its own token, so other tools and scripts don't use up the same hourly limit.
  • Use the Repositories field to sync only the repositories you need.
  • Choose a sync frequency that fits the size of your organization. Each sync makes at least one request per repository for each selected table, even when nothing changed.

Limitations

  • commits covers each repository's default branch. Commits on other branches appear in pull_request_commits once they are part of a pull request.
  • GitHub returns at most 250 commits for a single pull request in pull_request_commits.
  • A workflow run re-run more than three days after it was created is only updated by a full re-sync of the workflow_runs table.
  • When a label or assignee is removed from an issue or pull request, the old row stays in its table until a full re-sync of that table.
  • team_members includes members of child teams under their parent team, as GitHub lists them.
  • Pull requests don't include additions, deletions, changed files or a merged flag. A pull request is merged when merged_at is set.

Troubleshooting

"GitHub rejected the access token." The token is wrong, expired or revoked. Create a new one and update it on the connection.

"The token cannot read the organization …" or "… the repository …" Check the spelling, and that the token can reach it: for a fine-grained token, the resource owner and repository access; for a classic token, the scopes and SSO authorization.

A table is empty for some repositories. Repositories with issues turned off, without GitHub Actions, or without commits have no rows in the matching tables. They are skipped and don't fail the sync.

The teams or member tables fail or come back empty. The token needs the organization Members permission (fine-grained) or the read:org scope (classic).

Was this page helpful?