Scripts in this directory ran once (or a small number of times) to import
or repair specific data sets and are kept for historical reference rather
than active use. Each filename links back to the file's git history —
git log --follow shows the full context.
| Script | Last touched | What it did |
|---|---|---|
import_dreamflows.py |
2026-04-11 | Scraped Dreamflows run pages and linked them back to reach rows by AW ID so their text descriptions could be surfaced alongside live flows. |
fix_sort_names.py |
2026-04-11 | Backfilled reach.sort_name for older rows where the value was missing or inconsistent with the current naming convention. |
link_ok_guidebook.py |
2026-04-11 | Populated reach_guidebook rows for the Oregon "Paddling Oregon" / "Soggy Sneakers" entries from a flat source list. |
backfill_gauge_huc.py |
2026-04-23 | Filled gauge.huc for existing rows by joining gauge coordinates against the WBD HUC12 polygons. New gauges get HUCs at insert time via levels assign-huc. |
backfill_gauge_metadata.py |
2026-04-19 | One-shot enrichment of gauge.river / gauge.location / gauge.elevation from the agency-metadata caches; the same resolver runs incrementally in seed_gauge_display.py for new gauges. |
backfill_gauge_state.py |
2026-04-25 | Backfill for migration 0010 (added gauge.state). Tier 1: usgs_site.state_cd from the gauges metadata cache; tier 2: distinct state of any linked reach. New gauges get state at write time. |
recompute_midpoints.py |
2026-04-20 | One-shot recompute of reach.latitude / reach.longitude as the arc-length midpoint of reach.geom (vs. the old straight-line midpoint of put-in/take-out). Subsequent geom updates compute the midpoint at write time. |
install-editor-feature.sh |
2026-04-12 | Phase 1 sudo install steps for the editor / login feature (nginx zones, PHP-FPM pool, secrets dir). Subsumed by hardening/install.sh and the per-deploy nginx snippets under deploy/. |
split_usgs_sources.py |
2026-05-05 | Gave every gauge with a usgs_id a dedicated USGS source named with the digit station ID (RENAME for 8 already-USGS sources, INSERT for 69 NWS-only gauges). Stops fetch-usgs-ogc from dumping USGS observations into NWS source rows where _build_site_map would otherwise fall back. |
backfill_usgs_nwis.py |
2026-05-05 | Companion to split_usgs_sources.py — pulled 90 days of NWIS IV history for the freshly-split USGS sources so the new series had continuity from day one. 155 of 166 gauges returned data; 11 stations are inactive in NWIS. |
import_aw_usgs_reaches.py |
2026-04-22 | Three-phase import of American Whitewater reaches keyed by USGS gauge: created missing gauge + source + gauge_source rows, inserted reach rows for unmatched AW reaches, and fetched AW geometry via the GraphQL API. Standalone (stdlib only). Companion to match_aw_reaches.py. |
match_aw_reaches.py |
2026-04-22 | Matched American Whitewater reaches to local reaches via shared gauge source IDs. Cached AW reaches per state into gauges.db, then processed the cache to update reach.aw_id, put-in/take-out coordinates, and backfill missing usgs_id / cbtt_id. Output also drove the now-deleted docs/wgb-matches.md. |
install-observability.sh |
2026-05-02 | Installed the nginx timed-log-format drop-in, the CSP violation report sink, and the logrotate config for the CSP log. Enabled per-route latency quantiles and CSP violation diffs for /home/pat/logs.analyze/analyze.py. Backups land under /var/backups/kayak-nginx/. |
fix_reach_class_data_type.py |
2026-05-09 | Walked reach_class rows where low_data_type='flow' but the stored value (typically < 30) looked like a gage height in feet rather than CFS — proposed flipping data_type from flow to gauge on each affected field. Per-row y/n/q acceptance, with a kayak.db snapshot before any writes. |
apply-audit-timer-biweekly.sh |
2026-05-11 | One-shot patch that installed the bi-weekly kayak-audit-gauges systemd schedule on the live server (commit 972f1d6). |
apply-nginx-favicon.sh |
2026-05-11 | One-shot patch that installed the /favicon.ico location block from conf/levels.nginx into live nginx (commit 49b2321). |
fetch_dem_tiles.py |
2026-05-23 | Downloaded the 3DEP/OPR DEM GeoTIFF tiles covering all reach geometries into DEM-cache/ (gitignored) for the gradient-profile pipeline. |
sample_reach_elevations.py |
2026-05-23 | Sampled per-vertex elevations along each reach.geom from the local DEM tiles into Elevation-cache/ (gitignored), feeding the gradient computation. |
compute_reach_gradient.py |
2026-05-23 | Computed reach.max_gradient + the binned reach.gradient_profile JSON from the sampled elevations; result captured in migration 0046. |
emit_max_gradient_migration.py |
2026-05-23 | Emitted migration 0046 (per-row UPDATEs of max_gradient/gradient_profile) from the local DB; byte-stable regeneration. |
snap_reach_to_channel_min.py |
2026-05-23 | Prototype/diagnostic: snapped canyon-reach geom vertices to the perpendicular channel-floor minimum to remove NHD-trace artifacts (migrations 0052/0054/0055/0056). |
Small self-contained side projects kept for context. Each has its own README describing the experiment and how to revive it.
| Subdirectory | Last touched | What it was |
|---|---|---|
map-color-tune/ |
2026-05-11 | Standalone three-pane Leaflet page that renders the same reach traces over OpenTopoMap, OpenStreetMap, and Esri Satellite so trace colors / casing / line-weight can be A/B'd against every basemap at once. Used while tuning the production static/map.js palette. See subdir README for redeploy steps. |
regressions-WhiteSalmon/ |
2026-03-01 | R lm() calibration of White Salmon at Husum (gauge) vs. White Salmon at Underwood (feet) — predates the calc-expression / rating-table flow. Inputs: data (tabular), outputs: fit.pdf. Rerun with R --no-save < regress.R from the subdir. |
You can still run these from the repo root via:
/home/pat/.venv/bin/python3 docs/one-offs/<script>.pybut review the code first — each assumes a specific state of the live DB and external data that may have shifted since the last run.
- Write the script under
scripts/while iterating. - Once it has run and the outcome is captured in the DB, move it here
with
git mvand add a row to the table above.