Add StalledDiskPrimary analysis and recovery by vtorc #16050
Conversation
joekelley
requested review from
deepthi,
GuptaManan100,
rohit-nayak-ps and
shlomi-noach
as code owners
Review ChecklistHello reviewers! 👋 Please follow this checklist when reviewing this Pull Request. General
Tests
Documentation
New flags
If a workflow is added or modified:
Backward compatibility
|
vitess-bot
bot
added
NeedsBackportReason
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #16050 +/- ##
==========================================
+ Coverage 68.23% 68.59% +0.36%
==========================================
Files 1541 1545 +4
Lines 197254 198060 +806
==========================================
+ Hits 134597 135864 +1267
+ Misses 62657 62196 -461 ☔ View full report in Codecov by Sentry. |
GuptaManan100
added
Type: Enhancement
| // Return error if the disk is stalled or rejecting writes. | ||
| // Noop by default, must be enabled with the flag "enable_stalled_disk_check". | ||
| if tm.fsManager.IsDiskStalled() { | ||
| return nil, errors.New("stalled disk") | ||
| } |
There was a problem hiding this comment.
I think it's better to have a boolean field or an error field here. Even if the disk is stalled, we do get all the other field information back in FullStatus that can be used. Also, an error in full status indicates to vtorc that it couldn't reach the vttablet.
There was a problem hiding this comment.
I developed this change by using fsfreeze -f /vt on the primary instance in a test keyspace. In my testing before making any code changes, I found that vtorc's invocation of FullStatus would timeout and return a context deadline exceeded error. Simple queries like select @@global.server_id; would hang until the filesystem was unfrozen.
We could add a boolean or error field to model this, but I think we'd want to return here either way if the check fails, and it seems a bit cleaner to me to return an error rather than a response message with mostly nil/zero values.
| if err != nil { | ||
| if config.Config.EnableStalledDiskPrimaryAnalysis && strings.Contains(err.Error(), "stalled disk") { | ||
| stalledDisk = true | ||
| } | ||
| goto Cleanup |
There was a problem hiding this comment.
If we don't send the stalled disk as an error back as suggested ☝️, we can set the value in the normal flow below and it would be written as part of mkInsertOdkuInstances, and we won't need to change UpdateInstanceLastChecked.
| type writeFunction func() error | ||
|
|
||
| func attemptFileWrite() error { | ||
| file, err := os.Create(path.Join(env.VtDataRoot(), ".stalled_disk_check")) |
There was a problem hiding this comment.
Are there any cases where VTDATAROOT isn't the same disk as MySQL datadir? For our deployment it IS the same disk - our datadir is one level below VTDATAROOT. But theoretically datadir could be on it's own disk 🤔
It might be more flexible to have a flag like --stalled_disk_check_dir / stalled_disk_check_root enable this feature vs a boolean "enable" flag
| @@ -105,6 +108,9 @@ func registerInitFlags(fs *pflag.FlagSet) { | |||
| fs.Var(&initTags, "init_tags", "(init parameter) comma separated list of key:value pairs used to tag the tablet") | |||
| fs.DurationVar(&initTimeout, "init_timeout", initTimeout, "(init parameter) timeout to use for the init phase.") | |||
| fs.DurationVar(&mysqlShutdownTimeout, "mysql-shutdown-timeout", mysqlShutdownTimeout, "timeout to use when MySQL is being shut down.") | |||
| fs.BoolVar(&enableStalledDiskCheck, "enable_stalled_disk_check", enableStalledDiskCheck, "if true, tablet will check if disk is stalled or rejecting writes") | |||
There was a problem hiding this comment.
I believe new flags need to use - instead of _. Correct @deepthi?
|
@joekelley this is awesome, thanks a lot for moving this forward! |
Description
At HubSpot we have had a handful of incidents where a primary becomes impaired due to disk issues. When this happens, we observe that vtorc assigns an UnreachablePrimary analysis and does nothing because the
FullStatuscall it makes to the tablet times out. We monitor for these cases outside of Vitess and resolve them by running ERS, but it would be ideal if vtorc could detect and address these cases itself.
This change adds support for a StalledDiskPrimary analysis and recovery by vtorc. To detect when a tablet has a stalled disk we add a
FileSystemManagerthat attempts to write a file to vt data root every five seconds and expose a method that the tablet manager invokes inFullStatusto report whether the disk is stalled. Vtorc is modified to check for thestalled diskerror fromFullStatusand record the result in theCleanupblock.We are in the process of testing a change to this effect in lower environments at HubSpot. We aren't running the latest version of Vitess, so our internal patch is a bit different than what is presented here and this exact implementation hasn't been tested. This is my first Vitess PR. Any and all feedback is greatly appreciated 🙂
Related Issue(s)
Slack discussion from this time last year: https://vitess.slack.com/archives/C02GSRZ8XAN/p1685456224040299
PR that came from that discussion but didn't get merged: #13207
Note that this implementation is heavily inspired by the comments on #13207.
Checklist
Deployment Notes