A while back we switched the tablet throttler from static configuration (command line flags) to topo-based configuration, which is dynamic.

However, heartbeat setup remains static, and depends on --heartbeat_enable and/or --heartbeat_on_demand_duration flags. These are set at tablet start time, and a singular HeartbeatWriter object is created based on those flags.

This is a problem, because it is then possible to:

1. Spin a tablet with no heartbeats.
2. vtctldclient ApplyThrottlerConfig --enable to enable the throttler.
3. Crickets.

The default throttler behavior is to read replication lag from _vt.heartbeat table. But if heartbeats are not enabled, then the throttler will never be able to work correctly. It will always reject requests.

What are the options

Enabling the throttler should always work and should not lead to a surprising scenario. We can solve it in two ways:

1. Always enable some form of heartbeat, even if not configured. Meaning, if --heartbeat_enable=false and --heartbeat_on_demand_duration=0 (or both just missing), then assume heartbeat_on_demand_duration=5s (5s just as an example). That was the intention behind Auto-set 5s on-demand heartbeat if --enable_heartbeat is disabled #15099, but see discussion in that PR.
2. Enable a heartbeat lease whenever the throttler requests heartbeats. Technically, this is mostly how on-demand heartbeats are implemented today: they're mostly dormant (they kick in once on startup but then go to sleep), and only wake when requested, where today the only entity that requests heartbeat lease is the tablet throttler.

Option (1) is really easy and requires near zero code changes: just ensure on-demand-duration is nonzero if heartbeats are otherwise completely disabled. However, it does introduce bit of an element of surprise.

Option (2) makes a lot of sense, causes less surprises, but requires a bit of a rewrite to the heartbeat writer, which now needs to be more dynamic.