You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description:
There are various situation where ES may reject the event with document already exist. Purpose of this issue to collect such cases and add a short documentation (under the whichever suitable place, in troubleshooting or support doc or es-output) as we are getting same question over and over.
Ingestion from agent
Possibly two cases I can think of now:
events are datastream with integration and integration has a fingerprint processor which sets the document _id. For example, tenable_sc integration may have logs-tenable_sc.vulnerability-{version} & logs-tenable_sc.plugin-{version} ingest pipelines which have fingerprint sets the _id:
Logstash is having a backpressure where it cannot acknowledge the events to agent, as a result agent timeouts and resends the event. In a reality events might be indexed already in the ES. Quick resolution would be extending agent timeout but may depend on the situation.
There is another potential cause for these 409 conflicts.
When integrations write to a TSDS enabled index, the document id is defined as "a hash of the document’s dimensions and @timestamp".
The document's dimensions are defined in the integration, and when events are sent at a frequency > 1 per millisecond, and the dimensions are insufficient to disambiguate those events, a version conflict will arise.
This has already been seen in the integrations for the elastic agent and mysql, and I suspect there are more that can cause the issue
Tell us about the issue
Description:
There are various situation where ES may reject the event with document already exist. Purpose of this issue to collect such cases and add a short documentation (under the whichever suitable place, in troubleshooting or support doc or
es-output
) as we are getting same question over and over.Possibly two cases I can think of now:
_id
. For example,tenable_sc
integration may havelogs-tenable_sc.vulnerability-{version}
&logs-tenable_sc.plugin-{version}
ingest pipelines which have fingerprint sets the_id
:Example log when Logstash receives a rejected event:
Logstash is having a backpressure where it cannot acknowledge the events to agent, as a result agent timeouts and resends the event. In a reality events might be indexed already in the ES. Quick resolution would be extending agent timeout but may depend on the situation.
etc.
URL:
Example: https://www.elastic.co/guide/en/logstash/current/introduction.html
Anything else?
The text was updated successfully, but these errors were encountered: