Each shard runs its own initdb. Results need to be consistent for things to work.
More specifically:
- initdb needs to produce the same results
- The last record produced by initdb needs to not be shard specific. Otherwise we end up with inconsistent last_record_lsn across shards.
We should re-use the initdb generated by shard 0 on other shards.
Two implementation paths:
- Shard 0 uploads initdb to s3, other shards poll S3
- Shard 0 is created first, storcon sets
existing_initdb_timeline_id in the timeline creation request for other shards which will prompt them to download instead of creating it inline.
There's bolder ways in which we could change initdb generation (do it on compute), but they're not in scope here.