bug: Configuration reload does not affect bucket configuration

[pedro-stanaka]

Bug description

What is happening?

1. I start the exporter with a mapping configuration (see example config at the end).
2. I send some samples of the metric constantly (before metric had chance to churn)
3. I change mapping configuration and update the bucked layout for my metrics
4. I trigger a reload via sighup/web lifecycle
5. After sending new samples, old bucket configuration is still present and not new, updated one.

This is happening because the metric is never deleted after the reload from the Registry.

What you expected to happen?

On step 5. I would expect the new bucket configuration to take place.

Notes

Example config:

defaults:
    ttl: 3m
    match_type: regex
    observer_type: histogram
    histogram_options:
        buckets: [0.1, 0.25, 150, 500, 900]
mappings:
    - match: \bexample_metric\b
      observer_type: histogram
      histogram_options:
        buckets: [10, 40, 50, 80, 100, 120, 150, 200, 250]
      name: example_metric
      labels:
        statsd_exporter_config_mapping: "true"

[matthiasr]

Interesting! I don't know what a good solution is here tbh.

The "easiest" way to fix this would be to clear the registry on reload, but I think that would be very very disruptive. OTOH, we can't really predict from the mappings alone which metrics need clearing. Maybe we would need to defer this decision until a sample is ingested into a metric with conflicting bucket configuration, but then that would un-align metric resets 🤔

[pedro-stanaka]

One team mate came up with a solution that whenever there is a hot-reload of the configuration we delete all metrics that are configured in the mapping file from the registry (for us, that is disruptive, but acceptable). We are going to test that and come back with results. Would the repo be interested in such solution?

[matthiasr]

Hmm, yes, but let's make it opt-in with a command line flag. I don't want to force the clearing on everyone since it's not usually necessary.