-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix a possible data race with rollback manager and plugin reload #19468
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Would it be simple to write a unit test that validates this fix? |
@raymonstah Good call! I didn't think it would be possible but your comment led me to look a little closer and I realized I could set the RollbackPeriod for the test to a small value. This in turn led me to discover that I was not using the appropriate lock to address the race condition. I have updated this PR to use the lock on the routeEntry instead of the Router. |
re.l.RLock() | ||
defer re.l.RUnlock() | ||
|
||
return re.backend |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it might be worthwhile to extract these lines to a helper method so that clients don't have to remember to place a lock when reading backend
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am sympathetic to this. However, I have a few concerns
- use of
routeEntry
all happens within thevault
package so to be sure that clients use the helper methods we would want to refactor this to another package and make the fields private so that access to the fields require getters that hold locks, but - this code is in the hot path of vault requests so I am hesitant to do too much refactoring without involving the vault Core team
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I will leave this as-is and reach out to Core to see what they think about any more changes to the routeEntry
) * fix data race on plugin reload * add changelog * add comment for posterity * revert comment and return assignment in router.go * rework plugin continue on error tests to use compilePlugin * fix race condition on route entry * add test for plugin reload and rollback race detection * add go doc for test
The data race occurs when
RouteEntry.backend
inMatchingBackend()
.reloadBackendCommon()
performs a WRITE onRouteEntry.backend