Bug Report: --enable-tx-throttler can lead to crash due to race in go/vt/throttle

[timvaillancourt]

Overview of the Issue

vttablet can crash due to concurrent iteration/writes to a map in go/vt/throttler/replication_lag_cache.go:

fatal error: concurrent map iteration and map write
goroutine 455 [running]:
vitess.io/vitess/go/vt/throttler.(*Throttler).MaxLag(0xc000bb4728?, 0xb94630?)
        vitess.io/vitess/go/vt/throttler/throttler.go:236 +0x9c
vitess.io/vitess/go/vt/vttablet/tabletserver/txthrottler.(*txThrottlerStateImpl).updateMaxLag(0xc0009f85a0)
        vitess.io/vitess/go/vt/vttablet/tabletserver/txthrottler/tx_throttler.go:399 +0x1b4
created by vitess.io/vitess/go/vt/vttablet/tabletserver/txthrottler.newTxThrottlerState in goroutine 99
        vitess.io/vitess/go/vt/vttablet/tabletserver/txthrottler/tx_throttler.go:321 +0x4f6

This crash takes a long time to occur in our production, usually after 40-90 minutes

When reading go/vt/throttler/replication_lag_cache.go the issue is clear, entries from type replicationLagCache struct has concurrent update/deletes in .add() and concurrent reads in .MaxLag()

Reproduction Steps

1. Start vttablet with --enable-tx-throttler and --healthcheck_interval 1s. More tablets increases the likelihood of a crash
2. Cause a high rate of traffic that will hit the Transaction Throttler, which will call .MaxLag
3. Wait for vttablet to crash with fatal error: concurrent map iteration and map write

Binary Version

v19, or in our case: v15 with all txthrottler backports from v16-v19. The go/vt/throttler code in question is the same

Operating System and Environment details

N/A

Log Fragments

No response