"Haproxy HTTP Slowing Down" uses max total time

[janhorstmann]

Hi,

currently the Haproxy HTTP Slowing Down rule uses the haproxy_backend_max_total_time_seconds metric, which is described as

# HELP haproxy_backend_max_total_time_seconds Maximum observed total request+response time (request+queue+connect+response+processing)

By that description the metric shows a monotonically increasing value of the, to this point, highest observed total time. The expression then calculates the average of this value. The resulting value does not show an average increase in request time as is stated in the description, since it may hold a peak value from long ago.

Maybe using

# HELP haproxy_backend_total_time_average_seconds Avg. total time for last 1024 successful connections.

would be a better metric for the intention of the alert rule? Although outliers tend to drive this value up. Using some sort of percentile would be better suited, but does not seem to be available in this case