Skip to main content
The GitHub integration lets Loop investigate failing traces alongside your source code. Loop can:
  • Find the code or prompt logic behind a trace failure.
  • Read pull requests, reviews, comments, and CI results.
  • Investigate CI failures using GitHub Actions logs and artifacts.
  • Write a fix and regression test, push a branch, and open a pull request.
In conversations, Loop acts through your connected GitHub account, subject to your permissions and the GitHub App installation’s access. Automations remain read-only.
The GitHub integration is in public preview and can change before reaching general availability.

Connect GitHub

You need:
  • The Manage settings organization permission. Organization owners have it by default.
  • Ownership of the personal GitHub account, or approval from a GitHub organization owner. Organization members can request approval during installation.
Code can reach people without GitHub access through shared threads and automation runs. Automations use the GitHub App installation’s credentials. Choose repositories and project membership with this access in mind.
1

Connect an installation

Go to Settings > Integrations and click Connect GitHub.In GitHub, authorize access. Back in Braintrust, choose how to connect:
  • Use an existing installation: Select the installation for the GitHub account or organization you want to connect, then click Connect. If it’s marked Requires GitHub org owner, a GitHub organization owner must connect it.
  • Connect another GitHub account or organization: Click Install in a new organization. If all listed installations are already connected, click Install in the notice instead.
If no installations are available, Braintrust opens GitHub’s installation page automatically. When installing the Braintrust GitHub App, select the repositories you want Loop to access.
Integrations shows Pending and Needs org owner approval.
  1. Click Copy approval link and share it with a GitHub organization owner. Owners can also click Approve [organization] on GitHub.
  2. After approval, reload Integrations to finish connecting.
2

Verify access and set the default

Back in Integrations, click the installation’s row. Under Repositories, confirm that the repositories you need are listed, then click Set as default and Save.The installation shows a Default badge. Projects use it unless they select a different installation. Organization-level Loop threads always use the default installation, regardless of attached projects or their settings.
3

Use the code in Loop

Start a new Loop thread in the project and ask a question that combines traces with source code:
If Loop asks you to authorize GitHub, connect your GitHub account to continue.Loop clones up to five authorized repositories into github/<owner>/<repo>. Ask it to clone others as needed. See checkout limitations.

Connect your GitHub account

Loop uses your connected GitHub account in conversations, subject to repository access and permissions. When Loop displays an Authorize GitHub card:
  1. Click Authorize.
  2. Complete the connection in the popup, using a browser signed in to your own Braintrust account. API keys and OAuth bearer tokens can’t complete it.
  3. Return to the thread. Loop continues automatically.
If your account is already connected, Loop continues without prompting. To switch accounts or refresh access, ask Loop to reconnect GitHub and follow the Authorize GitHub steps again. To stop a connection request, click Cancel. Loop won’t ask again unless you request it.
If the popup is blocked, allow popups and try again. If authorization expires, ask Loop to try again.If GitHub requires you to authorize single sign-on (SSO) for an organization’s repositories, ask Loop to reconnect GitHub, even if your account is connected. After you authorize, Loop retries the failed operation. Reconnecting doesn’t fix missing repository permissions or disabled writes. For those, see Troubleshoot a blocked push or pull request under Manage connections.

Change GitHub settings for a project

Projects inherit the organization’s default installation. With the Manage settings organization permission, you can select another installation or disable GitHub for a project. In your project, go to Settings > Integrations. The Project usage column marks the installation the project uses as In use.
  • Use a different installation: Click the installation’s row. Under Project usage, turn on Use this installation, then click Save.
  • Return to the organization default: Open the selected installation, turn off Use this installation, then click Save.
  • Turn GitHub off for the project: Click Disable GitHub in [project name]. Loop threads in the project don’t get repository access until you click Enable GitHub in [project name], which returns the project to the organization default.
If the installation selected for a project is disconnected, the project doesn’t fall back to the organization default. In Integrations settings, click Use default installation to return to the default. Start a new Loop thread after changing a project’s installation. Existing threads keep the repositories already cloned into their sandbox.

Investigate CI failures

Loop can find GitHub Actions runs and download their logs and artifacts to investigate a failure:
Downloads require repository read access through your connected GitHub account.

Write changes through Loop

Loop can implement a fix and regression test, push a branch, and open a pull request using your connected GitHub account. To make changes, you need:
  • Braintrust access: Read permission on the project, or organization membership for an organization-level thread.
  • GitHub access: Permission for the requested action from both your account and the GitHub App installation.
The Allow Loop to create PRs and Allow Loop to create PRs for this project settings do not block writes through your connected account. Use GitHub permissions and branch protection rules to control those changes.
Start a new Loop thread in the project and ask for a change:
If prompted, connect your GitHub account. Loop continues after you connect.

How write access works

Automation runs, including Patterns runs, are read-only. In interactive threads:
  • Actions: Loop can push code and make repository or organization REST requests, including pull request updates and reviews. Your account and the GitHub App installation must permit each action.
  • Repositories: Loop can write to repositories where your account has write access and the thread’s GitHub connection allows access, including repositories it hasn’t cloned.
  • Branches: Braintrust doesn’t restrict branch names or block default-branch or force pushes. Use GitHub branch protection rules or rulesets to restrict them.
  • Revoking access: Change GitHub permissions or remove repository access to stop writes. This does not undo completed operations or delete existing branches and pull requests.

Access and security

Loop uses your connected GitHub account for reads and writes in conversations. Repository access must be allowed by all three:
  • Your GitHub account
  • The GitHub App installation
  • The project’s GitHub connection, or the organization’s default installation for organization-level threads
In Braintrust, you need Read permission for project threads or organization membership for organization-level threads. The PR creation settings do not restrict writes through your connected account. See how write access works.
  • Credentials: GitHub tokens are not exposed to the model or sandbox.
  • Downloads: Actions log and artifact downloads recheck your repository read access. Requests to GitHub’s storage hosts do not include your GitHub credentials.
  • Cloned files: Background checkouts can use the GitHub App installation’s credentials. Your personal GitHub permissions do not restrict access to files already cloned into a sandbox.

Manage connections

In Integrations settings, click an installation’s row. Click Set as default to make it the default, or Remove as default to stop providing a default installation, then click Save.These changes affect organization-level threads and projects that use the organization default. Projects that selected a different installation are unchanged. Without a default, organization-level threads and projects without a selected installation can’t use GitHub.
Changing or removing the default does not erase repository files already cloned into an existing sandbox. See sandbox lifecycle for how sandbox files are retained.
To change which repositories Braintrust can access, click Manage on GitHub in an installation’s row in Integrations settings, or click the row and then Manage repository access on GitHub below the Repositories list.If Manage on GitHub doesn’t appear, for example when Braintrust can’t reach GitHub to check the installation, open the Braintrust GitHub App on GitHub and click Configure to change its repository access.
Disconnecting an installation removes access through it for every Braintrust project that uses it. Projects that selected that installation do not fall back to the organization default. Disconnecting does not delete existing sandbox checkouts, or branches and pull requests Loop already created.
To remove an installation from Braintrust, click Disconnect installation in its row, or click the row and then Delete. Confirm the disconnection when prompted. The app remains installed on GitHub.
In the project’s Integrations settings, check that the installation marked In use is active and that the repository appears in its Repositories list. If the repository is authorized but was not automatically cloned, ask Loop to clone it by its owner/repo name.Start a new thread after changing the connection so Loop creates a new sandbox and prepares the checkouts. A failed checkout does not prevent the rest of Loop from working. Ask Loop to check whether the repository is available before continuing a source-code task.
When Braintrust blocks a push or pull request, Loop reports the reason in the thread and what to change. Common causes:
  • Insufficient permissions: Check that you have Read permission on the project and that your connected GitHub account and the GitHub App installation permit the operation. GitHub branch protections still apply.
  • The thread is organization-level: Check that you are a member of the Braintrust organization and that the repository is in its default installation.
  • Your GitHub account isn’t connected: Connect your GitHub account when Loop prompts you.
  • The run is an automation: Automation runs, including Patterns runs, are read-only. Ask for the change in an interactive thread.
  • The repository isn’t part of the project’s GitHub connection: Ask someone with the Manage settings organization permission to add the repository to the installation the project uses. Repository access is changed on GitHub, which can require a GitHub organization owner.

Limitations

  • Up to five automatic checkouts. Loop automatically clones the first five authorized repositories, ordered by GitHub repository ID. Ask Loop to clone additional authorized repositories.
  • Scoped REST access. Loop can use REST endpoints under /repos/<owner>/<repo>/ for connected repositories and /orgs/<owner>/ for their owning organizations, subject to GitHub permissions. GraphQL is not supported.
  • Shallow checkouts. Each checkout starts with the latest default-branch commit, without tags or earlier history. Ask Loop to fetch additional history or updates as needed.

Next steps