Skip to content

livenson/mxmap

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

129 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MX Map — Email Providers of Baltic Municipalities

Nightly

An interactive map showing where municipalities in Estonia, Latvia, and Lithuania host their official email — whether with US hyperscalers (Microsoft, Google, AWS) or Baltic/EU providers or self-hosted solutions.

View the live map

Screenshot of MX Map

How it works

The data pipeline has three steps:

  1. Preprocess — Loads ~182 Baltic municipalities from curated seed data, performs MX and SPF DNS lookups on their official domains (with domain guessing for missing entries), and classifies each municipality's email provider.
  2. Postprocess — Applies manual overrides for edge cases, retries DNS for unresolved domains, checks SMTP banners of independent MX hosts for hidden providers, then scrapes websites of still-unclassified municipalities for email addresses.
  3. Validate — Cross-validates MX and SPF records, assigns a confidence score (0–100) to each entry, and generates a validation report.
flowchart TD
    trigger["Nightly trigger"] --> seed

    subgraph pre ["1 · Preprocess"]
        seed[/"Seed data (EE, LV, LT)"/] --> fetch["Load ~182 municipalities"]
        fetch --> domains["Extract domains +<br/>guess candidates"]
        domains --> dns["MX + SPF lookups<br/>(3 resolvers)"]
        dns --> spf_resolve["Resolve SPF includes<br/>& redirects"]
        spf_resolve --> cname["Follow CNAME chains"]
        cname --> asn["ASN lookups<br/>(Team Cymru)"]
        asn --> autodiscover["Autodiscover DNS<br/>(CNAME + SRV)"]
        autodiscover --> gateway["Detect gateways<br/>(SeppMail, Barracuda,<br/>Proofpoint, Sophos ...)"]
        gateway --> classify["Classify providers<br/>MX → CNAME → SPF → Autodiscover → SMTP"]
    end

    classify --> overrides

    subgraph post ["2 · Postprocess"]
        overrides["Apply manual overrides"] --> retry["Retry DNS<br/>for unknowns"]
        retry --> smtp["SMTP banner check<br/>(EHLO on port 25)"]
        smtp --> scrape_urls["Probe municipal websites<br/>(/kontaktid, /kontakti, /kontaktai …)"]
        scrape_urls --> extract["Extract emails<br/>+ decrypt TYPO3 obfuscation"]
        extract --> scrape_dns["DNS lookup on<br/>email domains"]
        scrape_dns --> reclassify["Reclassify<br/>resolved entries"]
    end

    reclassify --> data[("data.json")]
    data --> score

    subgraph val ["3 · Validate"]
        score["Confidence scoring · 0–100"] --> gwarn["Flag potential<br/>unknown gateways"]
        gwarn --> gate{"Quality gate<br/>avg ≥ 70 · high-conf ≥ 80%"}
    end

    gate -- "Pass" --> deploy["Commit & deploy to Pages"]
    gate -- "Fail" --> issue["Open GitHub issue"]

    style trigger fill:#e8f4fd,stroke:#4a90d9,color:#1a5276
    style seed fill:#e8f4fd,stroke:#4a90d9,color:#1a5276
    style data fill:#d5f5e3,stroke:#27ae60,color:#1e8449
    style deploy fill:#d5f5e3,stroke:#27ae60,color:#1e8449
    style issue fill:#fadbd8,stroke:#e74c3c,color:#922b21
    style gate fill:#fdebd0,stroke:#e67e22,color:#935116
Loading

Quick start

uv sync

uv run preprocess
uv run postprocess
uv run validate

# Serve the map locally
python -m http.server

Development

uv sync --group dev

# Run tests with coverage
uv run pytest --cov --cov-report=term-missing

# Lint the codebase
uv run ruff check src tests
uv run ruff format src tests

Attribution

This project is a fork of mxmap.ch by David Huser, which maps email providers of Swiss municipalities. Adapted for the Baltic states (Estonia, Latvia, Lithuania) with Baltic-specific provider detection (Telia, TET, Zone.eu, Baltic ISPs), curated seed data, and Baltic municipality geodata.

Related work

Contributing

If you spot a misclassification, please open an issue with the municipality ID and the correct provider. For municipalities where automated detection fails, corrections can be added to the MANUAL_OVERRIDES dict in src/mail_sovereignty/postprocess.py.

About

Email providers of Baltic municipalities displayed on a map

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages

  • Python 78.5%
  • HTML 21.5%