[FR] Add ability to use custom fields in monitoring API

[garethhumphriesgkc]

The monitoring API is great for getting visibility into what logstash is doing at a node or pipeline level, however it's very opaque at the layer of the actual event data. For example, if I see a logstash node suddenly go slow or start doing a lot of GC, all I know is what node or pipeline that happened in - if I have multiple data sources going through that pipeline (e.g., lots of servers sending syslog data), there's no way to know which source caused it.

Arguably it isn't logstash's job to think about the data, and there are better tools for analysing the data itself in cases when that can help, but in the case for example of logstash CPU going through the roof for a period of time, it can very difficult to know which events caused the issue.

I think it would be useful to be able to break down monitoring data by one or more fields in the event itself when calling the /_node/stats/* endpoints. I would like to see a start up parameter with a list of fields in the event data to aggregate data by (e.g. api.data.group_by: ["hostname"]), and then be able to pass a GET parameter akin to ?group_by=hostname which would return the same results the API normally does but aggregated by the events' value of hostname. This would allow users to identify key fields they would like to understand at the logstash level ahead of time and collect data broken down by it.

There would of course of increased storage requirements, so there would have to be warnings around how to use, and possibly config parameters limiting the number of fields and/or values to aggregate by.