Creating custom scheduled tasks and digests requires Starter with on-demand usage enabled, Pro, or Enterprise. Without it, you can still edit or pause existing automations and create Patterns rules.
Create a digest
Digests summarize project activity on a schedule. They can report changes in quality, cost, or user behavior, with links to supporting traces. Unlike pattern alerts, a Digest sends a scheduled summary even when there are no significant changes.- Go to Settings > Digests and click Digest.
- Enter a Name and optional Description.
- In Agent instruction, describe what to summarize, or select a suggested prompt and adapt it.
- Choose a Writing style: Compact (the default) for metrics, highlights, and trace links, or Detailed for more explanation and next steps.
- Set the Schedule to determine how often the Digest runs and how far back it looks.
- Under Destinations, add at least one Slack channel, PagerDuty integration, or webhook.
- Review Preview final prompt, then click Create.
Create a scheduled task
Scheduled tasks run custom investigations with a schedule, query range, and permission to make specific changes.- Go to Settings > Scheduled tasks and click Scheduled task.
- Enter a Name and optional Description.
- In Agent instruction, describe what to investigate and what to return. Use the recipes for examples.
- Set the Schedule to determine how often the task runs.
- Set Default query range to determine how far back each run looks, up to 30 days. Each run examines the full range, including previously analyzed traces.
-
Under Write tool permissions, select only the tools the task needs to create or edit objects. Reading project data doesn’t require approval.
Permissions and ongoing effects
- Braintrust objects: Enable the corresponding write tools. Pattern rules can create and update patterns.
- GitHub repositories: Runs can read code, pull requests, and CI checks from connected repositories. GitHub writes require an interactive thread.
- Online scoring rules: Runs can create or update rules when permitted. These rules score production traffic continuously, including after the automation run ends.
- Created objects: Objects aren’t linked to any pattern the run recorded. If a pattern’s monitor chart measures a scorer or facet the run created, deleting that object stops new data from reaching the chart.
- Optionally add a Slack channel, PagerDuty integration, or webhook under Destinations. Use Formatting prompt to customize the message or webhook JSON.
- Click Create, then Run now to test the task and review its results in Past runs.
Run an automation now
A manual run lets you test an instruction or get results without waiting for the schedule. The next scheduled run still happens at its usual time. You need project Update permission, and you can’t run a paused automation. To run a saved automation, open it and click Run now. What happens depends on whether the automation is continuous:- Without continuous mode: Braintrust starts a one-off run in a new thread. The run covers the most recent query range, ending at the current time minus any evaluation delay.
- With continuous mode: Loop continues the current thread if more than 1,000 new traces have arrived since its last analysis. Otherwise, or if there’s no current run, Braintrust starts a full run in a new thread. Only traces matching the automation’s filter, if set, count toward the threshold. If a run is already in progress, nothing new starts.
How continuous checks work
How continuous checks work
Continuous, off by default, checks for new traces between scheduled runs. Enable it in Schedule. To turn it off, choose a schedule option again. Your existing schedule still controls full runs, each in a new thread.Braintrust checks every 45 to 75 minutes. If more than 1,000 new traces have arrived since the last analysis, Loop analyzes them in the current run’s thread and sends only significant new findings to your destinations, not a full report. For a pattern rule with a Traces filter, only matching traces count toward the threshold.Check timing depends on the schedule:
- Daily, Weekly, or Custom (cron): Checks run for 12 hours each day, starting at the time of day of the next scheduled run in the schedule’s timezone. A weekly Monday run at 9:00 AM gets checks from 9:00 AM to 9:00 PM every day between full runs.
- Interval over 1 hour: Checks run at any time of day.
- Interval of 1 hour or less: No continuous checks.
run_scheduled_loop_job tool on the Braintrust MCP server. It works like Run now: it requires project Update permission, and it can’t run a paused automation or start the run in a new thread. The tool only confirms that the run was submitted, so review the output on the Past runs tab.
Loop automation recipes
Use Digests for recurring summaries or scheduled tasks when you need custom query ranges or write permissions. Each recipe identifies its type and includes settings and an instruction you can adapt.Daily activity digest
Daily activity digest
Sends a brief summary of recent activity and notable problems to Slack.
Agent instruction
Weekly user-friction digest
Weekly user-friction digest
Summarizes where users struggle, with evidence and suggested follow-up work.
Agent instruction
Weekly cost report
Weekly cost report
Posts a cost breakdown to Slack every Monday morning.
Instruction
Daily quality review
Daily quality review
Compares existing quality scores with the previous day and investigates possible regressions. This example uses
scores.factuality, with higher values indicating better quality. Replace it with a score your project already records and adjust the direction if lower values are better.Instruction
Recent error summary
Recent error summary
Summarizes errors in the same subset of logs you monitor on a dashboard. Replace
metadata.environment = 'production' with the filter conditions from your dashboard. Include the conditions in the instruction so the run has the scope it needs.Instruction
Focused pattern discovery
Focused pattern discovery
Narrows pattern discovery to one concern, using an existing facet to choose which traces to investigate. Run this alongside the default Pattern discovery rule when a specific area deserves its own cadence.
Instruction
Review past runs
Open the automation, select Past runs, and select a run to review the work Loop did.- Run history: Each run creates a read-only thread that stays with its automation. These threads don’t appear in the Loop page’s thread list and can’t be deleted like interactive threads. Coding agents can list these runs and their links with the
list_scheduled_loop_job_runstool on the Braintrust MCP server. - Follow-up questions: Select Fork to chat to copy the transcript into a private, editable thread where you can investigate the run. The fork doesn’t send reports to the automation’s destinations, but patterns it creates or reopens can trigger project alerts.
- Failed runs: Select Retry & continue to resubmit the latest failed turn in the same thread. This action applies to a failed automation run, not a delivery request from an interactive fork. Retry requires project Update permission and expires four days after the run. It uses the run’s original settings and saved destinations, even if you’ve changed the automation.
Pause an automation
Pause an automation to stop its scheduled runs while keeping its configuration and history. Do this when you’re tuning an instruction, controlling spend, or investigating a run that behaved unexpectedly. Objects its runs already created stay in your project. To pause, open the automation and turn off Active. The automation then shows as Paused on its Scheduled tasks or Digests Settings page. Turn Active back on to resume the schedule. Automatic pauses. An automation that uses a built-in model, or has Auto on, pauses automatically when it can’t run. The automation list marks it with a warning indicator, and opening it shows a banner with the reason:- Default provider unavailable: The automation has Auto on and the default provider can’t serve it. Click Configure default to update the default, or turn off Auto and choose another provider.
- Credit limit reached: Your model credits are depleted. The automation doesn’t switch to your own AI providers on its own. Choose another provider or add a payment method.
- Provider unavailable: The selected built-in model is unavailable, or built-in models are disabled for your organization. Click AI provider settings to review your providers, or Add provider if your organization has no configured providers.
Delete an automation
To delete an automation, open its Scheduled tasks or Digests Settings page and delete it from the list. Deleting an automation doesn’t affect the objects its runs already created. Patterns, scorers, facets, and other objects a run produced remain in your project, and any online scoring rules a run created keep running until you stop them separately.Next steps
- Enable Patterns for scheduled pattern discovery.
- Set up alerts for conditions you can already define, without an agent run.
- What Loop can do to decide what’s worth automating.