Skip to content

feat(findall): show cost and enrichments before a run, add --no-enrich and preview - #166

Open
vidiyala99 wants to merge 1 commit into
parallel-web:mainfrom
vidiyala99:findall-cost-visibility
Open

vidiyala99 wants to merge 1 commit into
parallel-web:mainfrom
vidiyala99:findall-cost-visibility

Conversation

@vidiyala99

Copy link
Copy Markdown

I ran my first parallel-cli findall run with the defaults (core, -n 10) on a new account. The published formula says that's at most $2.00 + 10 x $0.15 = $3.50. The run was billed $3.75, and the Usage page showed "Task API v1: 11" on a day I never called the Task API.

The difference is enrichments. run_findall applies whatever enrichments ingest suggests (enrich=True), each one billed per match at its Task processor rate. For my objective ingest added a founding_date enrichment on core, so 10 x $0.025 = $0.25. The CLI never mentions this, and there's no way to turn it off. --dry-run doesn't help either, because ingest runs again when the real run starts and isn't deterministic: the same objective came back with 0, 1, then 0 suggested enrichments across three calls.

At -n 10 it's a quarter. At -n 1000 with a couple of core enrichments it's an extra $50 nobody asked for.

What this changes, all in findall run:

  • As soon as ingest returns, it prints the enrichments the run will add (field names and processor) and an estimated max cost, e.g. Est. cost: up to $3.75 ($2.00 + 10 matches x $0.175). The same estimate is in --dry-run, in both text and --json (estimated_cost, apply_enrichments).
  • --no-enrich skips suggested enrichments.
  • -g preview is back for findall run. The docs say to "Always run with generator: "preview" first", but the CLI only accepted base|core|pro. Preview was removed in feat: add FindAll to DuckDB integration + new CLI commands #58 because it doesn't support enrichment reliably, so it stays out of FINDALL_GENERATORS (the DuckDB paths are unchanged), enrichment is always skipped for it, and --match-limit must be 5-10.
  • Fixes --dry-run printing enrichments as - : . It read name/description keys that ingest enrichments don't have; it now lists the fields and processor.

The prices come from https://docs.parallel.ai/getting-started/pricing and live in FINDALL_GENERATOR_PRICING and TASK_PROCESSOR_PRICE_PER_RUN in core/findall.py, with estimate_findall_cost() and summarize_enrichments() alongside. An unknown generator or enrichment processor makes the estimate say "unknown" rather than guess. I know hardcoded prices can drift, so if there's an API or a better source for these I'm happy to switch to it. That would also help with #158 and parallel-sdk-python#17.

Live run with this branch:

$ parallel-cli findall run "Find AI web search API companies founded after 2022" -g preview -n 5
Starting FindAll with generator=preview, match_limit=5
Generator: 5-10 candidates - test a query cheaply before a full run

Schema ready - entity type: companies
  Enrichments skipped: 1
    - founding_date (processor: core, billed per match)
  Est. cost: up to $0.10 ($0.10 + 5 matches x $0.00)
...
Status: completed | generated: 5, matched: 3 (1m52s)

The same objective on the defaults took 5m20s and $3.75.

Testing: 25 new tests in tests/test_findall.py (cost estimates including the $3.75 case above, run_findall with enrich=False and preview, and the CLI flag, dry-run and validation paths). ruff check, ruff format --check, tach check and ty check are clean. Full pytest: 864 passed, 14 failed, and the same 14 fail on main on my Windows machine (Unix permission, symlink and DuckDB event loop tests), so they're unrelated.

I used AI assistance while working on this and reviewed and tested the change myself.

🤖 Generated with Claude Code

@vidiyala99
vidiyala99 requested a review from a team November 10, 2026 16:59
…h and preview

A default `findall run` (core, -n 10) can bill more than the published
formula because it applies whatever enrichments ingest suggests, one Task
run per match, without saying so. Ingest is not deterministic: the same
objective returned 0, 1, then 0 suggested enrichments, so `--dry-run`
can't promise what the real run does.

- Print the enrichments a run will add, and an estimated max cost, as soon
  as ingest returns (and in --dry-run, text and JSON).
- Add --no-enrich to skip suggested enrichments.
- Offer the preview generator on `findall run` only (5-10 candidates,
  $0.10). It stays out of FINDALL_GENERATORS since parallel-web#58 found it doesn't
  support enrichment reliably, and enrichment is always skipped for it.
- Fix dry-run listing enrichments as "- : " (it read name/description
  keys that ingest enrichments don't have).
- Add estimate_findall_cost() and summarize_enrichments() to core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vidiyala99
vidiyala99 force-pushed the findall-cost-visibility branch from 1d1d126 to dd880bf Compare November 10, 2026 17:06

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant