-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use a small pool of workers to run postUnsealFuncs in parallel #18244
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
} | ||
for i, v := range c.postUnsealFuncs { | ||
wg.Add(1) | ||
workerChans[i%postUnsealFuncConcurrency] <- v |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know a terrible amount about c.postUnsealFuncs
but if this variable contains truly independent functions that we can run in parallel, I wonder if it makes more sense to just loop over them and run each one in its own goroutine, vs having a worker pool with a static size.
My main concern with this particular piece is index wraparound when len(c.postUnsealFuncs) > postUnsealFuncConcurrency
. At that point, you'll be overwriting the contents of the workerChans
slice, no? Given the default size of 32, maybe the probability of this is low (I don't know what len(c.postUnsealFuncs)
typically is), but since the size is tunable, it could be anything.
Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The issue concretely is that we have some customers with 5k+ PKI mounts that need to be migrated when upgrading from <=1.10.x to 1.11+.
If we run 4k+ PKI mount migration in full parallel, we absolutely slam backend storage and also the Vault node's CPU, in addition to potentially having too many goroutines, thus starving Raft's heatbeats (if its in use).
Setting a relatively safe upper limit here is good I think. I'd like to maybe see 2x CPU count by default rather than a hard-coded default (just to prevent starvation on nodes with a single CPU core).
Each mount's migration/initialize func yes, is strictly independent, but the combination of work of all of them might be enough to bring down Vault if run in parallel.
Today, we run them strictly serially, which does avoid this problem, but means we strictly stall for IO, then do compute, then stall for more IO, &c.. --- and if each migration only takes 100ms, on 5k+ mounts, we're sitting north of 8 minutes to even unseal, which blows past the default context deadline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I'd thought about runtime.NumCPUS(), which I'm back to liking again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
vault/core.go
Outdated
for v := range workerChans[i] { | ||
v() | ||
func() { | ||
defer wg.Done() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We Add every time we send to a channel, so I agree we need to Done every time we read from it. Why the extra func/defer though? Couldn't we just have a loop body of { v(); wg.Done() }
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although alternatively, we could instead Add once per postUnsealFuncConcurrency and Done when the worker exits.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I suspect so. The defer would still let us Done in a panic, but we'd have bigger problems then.
* Initial worker pool * Run postUnsealFuncs in parallel * Use the old logic for P=1 * changelog * Use a CPU count relative worker pool * Update vault/core.go Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com> * Done must be called once per postUnsealFunc * Defer is overkill Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
In further review with new understanding after #18244, loading configuration and CRLs within the backend's initialize function is the ideal approach: Factory construction is strictly serial, resulting in backend initialization blocking until config and CRLs are loaded. By using an InitializeFunc(...), we delay loading until after all backends are constructed (either right on startup in 1.12+, else during the initial PeriodicFunc(...) invocation on 1.11 and earlier). We also invoke initialize automatically on test Factory construction. Resolves: #17847 Co-authored-by: valli_0x <personallune@mail.ru> Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
* Move cert auth backend setup into initialize In further review with new understanding after #18244, loading configuration and CRLs within the backend's initialize function is the ideal approach: Factory construction is strictly serial, resulting in backend initialization blocking until config and CRLs are loaded. By using an InitializeFunc(...), we delay loading until after all backends are constructed (either right on startup in 1.12+, else during the initial PeriodicFunc(...) invocation on 1.11 and earlier). We also invoke initialize automatically on test Factory construction. Resolves: #17847 Co-authored-by: valli_0x <personallune@mail.ru> Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Add changelog entry Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> --------- Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> Co-authored-by: valli_0x <personallune@mail.ru>
Initially size 32, configurable via a VAULT_POSTUNSEAL_FUNC_CONCURRENCY
variable in case of trouble.