Repository navigation
Conversation
…ools with structured errors Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nknown, redact credential at output Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MCP closure slice (dispatch CX-20261014-02, Slice 1): an AI client can turn the Free starter into a useful mock, inspect and edit it safely, run the supported Sequence/Reset retry workflow, and see exact capture evidence — without hand-editing DB/UI. Plan: requestbin/requestbin
storage/reports/ai_friendly_mock_execution_plan_20261014.md(SaaS docs PR linked below).New tools:
get_mock_endpoint(rules in match order,updatedAttoken, saved/deployed/legacy state, warnings),update_mock_rule(existing rule incl. starter; requiredexpectedUpdatedAtCAS; static + Sequence; omit = keep,null= clear),reset_mock_sequence(202 = queued). Fixed:add_mock_rulesaid "higher priority first" (matcher is lower first) and advertised/api/v1/*(only exact path or*exist); Free "create → add rule" 403s, so the documented Free flow is create → inspect → update starter;deploy_mock/others no longer imply isolation. Errors are structured (plan_limit,conflictwith intended change + current state,validation,scenario_runtime_unavailable,not_found,network_error,ambiguous_writewith one read-only reconciliation, …); keys are redacted. Nothing deploys as a side effect; no flag/cap/quota change; no SaaS, Go or runner code change.Risk Level
Systems Affected
Dependencies — NOT production-ready by itself
Relies on SaaS behaviour that is still unmerged: rule PATCH with
expectedUpdatedAtCAS (requestbin#310/#313),sequencewrite +…/reset+publishedRulessnapshot (requestbin#311, flagENABLE_MOCK_SCENARIO), Go sequences/capacity (requestbin-mock#3), capture execution metadata (requestbin-runner#5). Against a server without them the tools degrade to explicit errors (scenario_runtime_unavailable, legacy state) — verified.Evidence (labels matter)
npm test: 14 tests through the real MCP protocol client (in-memory transport) with a faked HTTP layer — deterministic, not server evidence. Build + types pass.node dist/index.jsstdio → API → mock HTTP → persisted capture. Transcripts (keys redacted) indemo/transcripts/: Free loop (starter →POST /checkout500, real HTTP 500, capture matched rule, 1 rule throughout, add →plan_limit); Sequence 429 → client timeout (2 s) → 200 after 5 s, persisted run/step/outcome (response_written,client_cancelled), Reset queued, next request step 1 in a new run id; PRO shadowing/priority (equal priority loses to the catch-all;-1wins; glob stays literal), cap rejection at 5; conflict (+ repeat with old token), wrong account (not_found), invalid input, unreachable API, and a write applied then response lost (ambiguous_write, reconciledapplied, no duplicate).claude -p, MCP tools only) read the endpoint, edited the starter in place, deployed, reported — verified independently by real HTTP 500 + capture (demo/transcripts/llm-*.jsonl). N=1; not a success-rate claim.Quickstart for a ~5-minute reproduction is in README (acceptance target, not a measured activation figure).
Rollback: don't merge / revert; no data migration.
Update 2026-10-14 — contract-safety rework (CX-20261014-03, 742b6a4) — corrects claims above
Corrected claims. (1) "Against a server without these dependencies the tools degrade to explicit errors" was false for the edit path: origin/master SaaS returns 200 to a PATCH while ignoring
expectedUpdatedAtandsequence. (2) Thereconciled: applied/ "safe to retry once" ambiguous-write result was unproven. (3) "Keys are redacted" covered messages only, notintendedChange/current/details.Fixes. (1) Fail closed: capability-dependent writes (
update_mock_rule,add_mock_rulewith a sequence,reset_mock_sequence) are refused before sending (capability_unavailable/scenario_runtime_unavailable) unless the server itself states the capability via the explicitcapabilitiesobject (mock-rules-v1) — never inferred fromupdatedAt/publishedRules/versions/URLs. Read-only tools and a plain add still work on old servers. Requires the new SaaS contract: requestbin#317 (stacked on #313). (2) Ambiguous outcomes stay UNKNOWN: candidates only (withmatchesIntendedExactlyover full config incl. headers/sequence), never auto-retried; body-read failures, truncated/non-JSON 2xx, bad shapes and timeouts after a write are ambiguous; boundedREQUESTBIN_TIMEOUT_MS; deploy/reset/add wording no longer claims repeat-safety. (3) The configured key is scrubbed at the output boundary (nested, keys,intendedChange/current/details/success), output-only; scope is that key, not arbitrary secrets in captures.Evidence. Tests 14 → 32 (regressions written first, 16 red before the fixes), build/types pass. Live (real MCP client → API :3000 → mock :8081, synthetic users): old server refused the stale edit and the other writer's change was kept; capability-enabled server re-ran the Free/Sequence+Reset/PRO/failure matrix; ambiguous write →
outcome: unknownwith candidates, no duplicate; key placed in a rule never appears in tool output while the stored rule is unchanged. Transcripts:demo/transcripts/rework/. The earlier single LLM-client run predates the rework and is not repeated. Dependencies for production use: #317 + #313 (+ #310/#311, mock#3, runner#5 for Sequences). Not merged/published.