Skip to main content
Loop automations investigate project data on a schedule, such as changes in cost, quality, or errors. Choose a Digest for recurring summaries, a Scheduled task for a custom investigation, or a pattern rule to discover and maintain patterns. Send results to Slack, PagerDuty, or a webhook.
Creating custom scheduled tasks and digests requires Starter with on-demand usage enabled, Pro, or Enterprise. Without it, you can still edit or pause existing automations and create Patterns rules.

Create a digest

Digests summarize project activity on a schedule. They can report changes in quality, cost, or user behavior, with links to supporting traces. Unlike pattern alerts, a Digest sends a scheduled summary even when there are no significant changes.
  1. Go to Settings > Digests and click Digest.
  2. Enter a Name and optional Description.
  3. In Agent instruction, describe what to summarize, or select a suggested prompt and adapt it.
  4. Choose a Writing style: Compact (the default) for metrics, highlights, and trace links, or Detailed for more explanation and next steps.
  5. Set the Schedule to determine how often the Digest runs and how far back it looks.
  6. Under Destinations, add at least one Slack channel, PagerDuty integration, or webhook.
  7. Review Preview final prompt, then click Create.
To read a completed Digest or ask follow-up questions, review its past runs.

Create a scheduled task

Scheduled tasks run custom investigations with a schedule, query range, and permission to make specific changes.
  1. Go to Settings > Scheduled tasks and click Scheduled task.
  2. Enter a Name and optional Description.
  3. In Agent instruction, describe what to investigate and what to return. Use the recipes for examples.
  1. Set the Schedule to determine how often the task runs.
  2. Set Default query range to determine how far back each run looks, up to 30 days. Each run examines the full range, including previously analyzed traces.
  1. Under Write tool permissions, select only the tools the task needs to create or edit objects. Reading project data doesn’t require approval.
    Enabled Braintrust writes run without asking for approval. All tools from connected project MCP servers are also available without approval, including in pattern rules. Restrict their access through the shared API key or OAuth account’s permissions.Allowing online scoring rules can start ongoing scoring costs that continue until you stop those rules.
    • Braintrust objects: Enable the corresponding write tools. Pattern rules can create and update patterns.
    • GitHub repositories: Runs can read code, pull requests, and CI checks from connected repositories. GitHub writes require an interactive thread.
    • Online scoring rules: Runs can create or update rules when permitted. These rules score production traffic continuously, including after the automation run ends.
    • Created objects: Objects aren’t linked to any pattern the run recorded. If a pattern’s monitor chart measures a scorer or facet the run created, deleting that object stops new data from reaching the chart.
  1. Optionally add a Slack channel, PagerDuty integration, or webhook under Destinations. Use Formatting prompt to customize the message or webhook JSON.
  2. Click Create, then Run now to test the task and review its results in Past runs.

Run an automation now

A manual run lets you test an instruction or get results without waiting for the schedule. The next scheduled run still happens at its usual time. You need project Update permission, and you can’t run a paused automation. To run a saved automation, open it and click Run now. What happens depends on whether the automation is continuous:
  • Without continuous mode: Braintrust starts a one-off run in a new thread. The run covers the most recent query range, ending at the current time minus any evaluation delay.
  • With continuous mode: Loop continues the current thread if more than 1,000 new traces have arrived since its last analysis. Otherwise, or if there’s no current run, Braintrust starts a full run in a new thread. Only traces matching the automation’s filter, if set, count toward the threshold. If a run is already in progress, nothing new starts.
To open the run’s thread once it starts, go to the automation’s Past runs tab. To start a full run in a new thread for a continuous automation, click next to Run now and select Run in new thread. Later checks continue that new thread. This option appears only for continuous automations.
Continuous, off by default, checks for new traces between scheduled runs. Enable it in Schedule. To turn it off, choose a schedule option again. Your existing schedule still controls full runs, each in a new thread.Braintrust checks every 45 to 75 minutes. If more than 1,000 new traces have arrived since the last analysis, Loop analyzes them in the current run’s thread and sends only significant new findings to your destinations, not a full report. For a pattern rule with a Traces filter, only matching traces count toward the threshold.Check timing depends on the schedule:
  • Daily, Weekly, or Custom (cron): Checks run for 12 hours each day, starting at the time of day of the next scheduled run in the schedule’s timezone. A weekly Monday run at 9:00 AM gets checks from 9:00 AM to 9:00 PM every day between full runs.
  • Interval over 1 hour: Checks run at any time of day.
  • Interval of 1 hour or less: No continuous checks.
Continuous isn’t available for automations with a threshold condition.
Coding agents can also run a saved automation with the run_scheduled_loop_job tool on the Braintrust MCP server. It works like Run now: it requires project Update permission, and it can’t run a paused automation or start the run in a new thread. The tool only confirms that the run was submitted, so review the output on the Past runs tab.
A run started from the MCP server sends results to the automation’s configured destinations, such as Slack, webhooks, or PagerDuty. Calling the tool again to check progress can start another run and deliver duplicate results.

Loop automation recipes

Use Digests for recurring summaries or scheduled tasks when you need custom query ranges or write permissions. Each recipe identifies its type and includes settings and an instruction you can adapt.
Sends a brief summary of recent activity and notable problems to Slack.
Agent instruction
Summarizes where users struggle, with evidence and suggested follow-up work.
Agent instruction
Posts a cost breakdown to Slack every Monday morning.
Instruction
Compares existing quality scores with the previous day and investigates possible regressions. This example uses scores.factuality, with higher values indicating better quality. Replace it with a score your project already records and adjust the direction if lower values are better.
Instruction
Summarizes errors in the same subset of logs you monitor on a dashboard. Replace metadata.environment = 'production' with the filter conditions from your dashboard. Include the conditions in the instruction so the run has the scope it needs.
Instruction
Narrows pattern discovery to one concern, using an existing facet to choose which traces to investigate. Run this alongside the default Pattern discovery rule when a specific area deserves its own cadence.
Instruction

Review past runs

Open the automation, select Past runs, and select a run to review the work Loop did.
  • Run history: Each run creates a read-only thread that stays with its automation. These threads don’t appear in the Loop page’s thread list and can’t be deleted like interactive threads. Coding agents can list these runs and their links with the list_scheduled_loop_job_runs tool on the Braintrust MCP server.
  • Follow-up questions: Select Fork to chat to copy the transcript into a private, editable thread where you can investigate the run. The fork doesn’t send reports to the automation’s destinations, but patterns it creates or reopens can trigger project alerts.
  • Failed runs: Select Retry & continue to resubmit the latest failed turn in the same thread. This action applies to a failed automation run, not a delivery request from an interactive fork. Retry requires project Update permission and expires four days after the run. It uses the run’s original settings and saved destinations, even if you’ve changed the automation.

Pause an automation

Pause an automation to stop its scheduled runs while keeping its configuration and history. Do this when you’re tuning an instruction, controlling spend, or investigating a run that behaved unexpectedly. Objects its runs already created stay in your project. To pause, open the automation and turn off Active. The automation then shows as Paused on its Scheduled tasks or Digests Settings page. Turn Active back on to resume the schedule. Automatic pauses. An automation that uses a built-in model, or has Auto on, pauses automatically when it can’t run. The automation list marks it with a warning indicator, and opening it shows a banner with the reason:
  • Default provider unavailable: The automation has Auto on and the default provider can’t serve it. Click Configure default to update the default, or turn off Auto and choose another provider.
  • Credit limit reached: Your model credits are depleted. The automation doesn’t switch to your own AI providers on its own. Choose another provider or add a payment method.
  • Provider unavailable: The selected built-in model is unavailable, or built-in models are disabled for your organization. Click AI provider settings to review your providers, or Add provider if your organization has no configured providers.

Delete an automation

Deleting an automation is permanent. If you might want the automation again, pause it instead.
To delete an automation, open its Scheduled tasks or Digests Settings page and delete it from the list. Deleting an automation doesn’t affect the objects its runs already created. Patterns, scorers, facets, and other objects a run produced remain in your project, and any online scoring rules a run created keep running until you stop them separately.

Next steps