Log rotation is extra aggressive on removing older log files

[supahcraig]

It seems that the log rotation is not preserving multiple iterations of the log. Whenever the active log file reaches 10MB it rolls to a timestamped version, as expected. However, it appears to also delete any prior rotated log files, regardless of the rotate_max_age_days setting.

For example, using the config below the initial log file will be written to test.log. Once it rolls it will rename to something like test-2024-06-21T15-43-44.388.log.gz

When the new test.log hits 10 meg it will roll to a new file named like test-2024-06-21T16-57-54.152.log.gz. This is exactly as expected. However, the prior file (test-2024-06-21T15-43-44.388.log.gz) is now removed. Because you're probably going to ask.... no, the prior file is not rolled into the new gzip.

It could be that the trouble is somewhere in this general area: https://github.com/redpanda-data/benthos/blob/main/internal/log/logrus.go#L30

MaxBackups: 1,

Here is a simple config to demonstrate the issue.

input:
  generate:
    interval: 0.001s
    mapping: |
      root.ID = uuid_v4()
      root.Name = ["frosty", "spot", "oodles"].index(random_int() % 3)
      root.Gooeyness = (random_int() % 100) / 100

output:
  stdout: {}

logger:
  level: TRACE
  format: logfmt
  add_timestamp: true
  timestamp_name: ts
  file:
    path: test.log
    rotate: true
    rotate_max_age_days: 50000