Skip to content

feat: mock inspect / in-place edit / sequence / reset tools + structured errors (v0.4.0, unpublished) - #3

Open
cuongmx wants to merge 2 commits into
mainfrom
feat/mock-closure-slice
Open

cuongmx wants to merge 2 commits into
mainfrom
feat/mock-closure-slice

Conversation

@cuongmx

@cuongmx cuongmx commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

MCP closure slice (dispatch CX-20261014-02, Slice 1): an AI client can turn the Free starter into a useful mock, inspect and edit it safely, run the supported Sequence/Reset retry workflow, and see exact capture evidence — without hand-editing DB/UI. Plan: requestbin/requestbin storage/reports/ai_friendly_mock_execution_plan_20261014.md (SaaS docs PR linked below).

New tools: get_mock_endpoint (rules in match order, updatedAt token, saved/deployed/legacy state, warnings), update_mock_rule (existing rule incl. starter; required expectedUpdatedAt CAS; static + Sequence; omit = keep, null = clear), reset_mock_sequence (202 = queued). Fixed: add_mock_rule said "higher priority first" (matcher is lower first) and advertised /api/v1/* (only exact path or * exist); Free "create → add rule" 403s, so the documented Free flow is create → inspect → update starter; deploy_mock/others no longer imply isolation. Errors are structured (plan_limit, conflict with intended change + current state, validation, scenario_runtime_unavailable, not_found, network_error, ambiguous_write with one read-only reconciliation, …); keys are redacted. Nothing deploys as a side effect; no flag/cap/quota change; no SaaS, Go or runner code change.

Risk Level

  • Low (additive tools + error shape; existing bin tools unchanged; package version bumped to 0.4.0 in this PR only — NOT published)

Systems Affected

  • requestbin.saas (consumes its existing API only)
  • requestbin-runner
  • requestbin-mock
  • billing / auth / middleware

Dependencies — NOT production-ready by itself

Relies on SaaS behaviour that is still unmerged: rule PATCH with expectedUpdatedAt CAS (requestbin#310/#313), sequence write + …/reset + publishedRules snapshot (requestbin#311, flag ENABLE_MOCK_SCENARIO), Go sequences/capacity (requestbin-mock#3), capture execution metadata (requestbin-runner#5). Against a server without them the tools degrade to explicit errors (scenario_runtime_unavailable, legacy state) — verified.

Evidence (labels matter)

  • npm test: 14 tests through the real MCP protocol client (in-memory transport) with a faked HTTP layer — deterministic, not server evidence. Build + types pass.
  • Live, real stack (local review DB, SaaS :3000 from requestbin#315 branch, mock :8081, Redis 6391; synthetic test-owned users, no Clerk accounts): real MCP client → node dist/index.js stdio → API → mock HTTP → persisted capture. Transcripts (keys redacted) in demo/transcripts/: Free loop (starter → POST /checkout 500, real HTTP 500, capture matched rule, 1 rule throughout, add → plan_limit); Sequence 429 → client timeout (2 s) → 200 after 5 s, persisted run/step/outcome (response_written, client_cancelled), Reset queued, next request step 1 in a new run id; PRO shadowing/priority (equal priority loses to the catch-all; -1 wins; glob stays literal), cap rejection at 5; conflict (+ repeat with old token), wrong account (not_found), invalid input, unreachable API, and a write applied then response lost (ambiguous_write, reconciled applied, no duplicate).
  • Actual LLM client: one Claude Code run (claude -p, MCP tools only) read the endpoint, edited the starter in place, deployed, reported — verified independently by real HTTP 500 + capture (demo/transcripts/llm-*.jsonl). N=1; not a success-rate claim.
  • Not verified: Cursor/Windsurf runs, production, any user/revenue effect.

Quickstart for a ~5-minute reproduction is in README (acceptance target, not a measured activation figure).

Rollback: don't merge / revert; no data migration.


Update 2026-10-14 — contract-safety rework (CX-20261014-03, 742b6a4) — corrects claims above

Corrected claims. (1) "Against a server without these dependencies the tools degrade to explicit errors" was false for the edit path: origin/master SaaS returns 200 to a PATCH while ignoring expectedUpdatedAt and sequence. (2) The reconciled: applied / "safe to retry once" ambiguous-write result was unproven. (3) "Keys are redacted" covered messages only, not intendedChange / current / details.
Fixes. (1) Fail closed: capability-dependent writes (update_mock_rule, add_mock_rule with a sequence, reset_mock_sequence) are refused before sending (capability_unavailable / scenario_runtime_unavailable) unless the server itself states the capability via the explicit capabilities object (mock-rules-v1) — never inferred from updatedAt/publishedRules/versions/URLs. Read-only tools and a plain add still work on old servers. Requires the new SaaS contract: requestbin#317 (stacked on #313). (2) Ambiguous outcomes stay UNKNOWN: candidates only (with matchesIntendedExactly over full config incl. headers/sequence), never auto-retried; body-read failures, truncated/non-JSON 2xx, bad shapes and timeouts after a write are ambiguous; bounded REQUESTBIN_TIMEOUT_MS; deploy/reset/add wording no longer claims repeat-safety. (3) The configured key is scrubbed at the output boundary (nested, keys, intendedChange/current/details/success), output-only; scope is that key, not arbitrary secrets in captures.
Evidence. Tests 14 → 32 (regressions written first, 16 red before the fixes), build/types pass. Live (real MCP client → API :3000 → mock :8081, synthetic users): old server refused the stale edit and the other writer's change was kept; capability-enabled server re-ran the Free/Sequence+Reset/PRO/failure matrix; ambiguous write → outcome: unknown with candidates, no duplicate; key placed in a rule never appears in tool output while the stored rule is unchanged. Transcripts: demo/transcripts/rework/. The earlier single LLM-client run predates the rework and is not repeated. Dependencies for production use: #317 + #313 (+ #310/#311, mock#3, runner#5 for Sequences). Not merged/published.

cuongmx and others added 2 commits October 14, 2026 22:57
…ools with structured errors

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nknown, redact credential at output

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant