Promitor throwing error msg="Error on ingesting samples that are too old or are too far into the future" in Prometheus logs

[SurveySky]

Hello,

I was hoping if you could help me identify the problem causing msg="Error on ingesting samples that are too old or are too far into the future" error. I'm not quite sure if this can be fixed through Promitor configuration or Prometheus.

Discovery helm chart version 0.8.1
Scraper helm chart version 2.9.0

I have done quite a lot of research regarding this error message, but nothing has helped so far.

My configuration is nothing special both on discovery and scrapers (I have spilt up agent scrapers based on resource), but I do have enableMetricTimestamps: false, I use metric relabeling and I have setup HA Prometheus, if this has any relevance. If you need any more information about my configuration, let me know.

I do have one suspicion that this might happen, because timestamps are off and Promitor does not remove deleted targets, but then I would expect all scrapers to throw this error and not just one or two. Also, after restarting the pod that throws error, error disappears for some time, and then another pod starts throwing errors.

Full error message: caller=scrape.go:1691 level=warn component="scrape manager" scrape_pool=serviceMonitor/promitor/promitor-agent-scraper-web-app/0 target=http://10.244.10.45:5000/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3

[tomkerkhove]

Do you have an example of the exposed metrics by Promitor?

I've never seen this before though, to be honest.

[SarunasZaleckis]

It does seem odd that the Helm Chart defaults defined as enableMetricTimestamps: true, whilst the Defaults file for the software defines it as false by default

[SurveySky]

Do you need anymore information about this? Or should I move this issue to Prometheus? @tomkerkhove

[tomkerkhove]

It does seem odd that the Helm Chart defaults defined as enableMetricTimestamps: true, whilst the Defaults file for the software defines it as false by default

I don't see a problem with that.

As mentioned, this error is visible only in Prometheus logs, but I see that Promitor agent scraper pod is causing the issue.
An example of metrics exposed by promitor-agent-scraper-web-app:

What exactly is the problem? I did not check the timestamps but is it because they are in the future or? Maybe you're facing clock skew.

[SurveySky]

Timestamps are removed from Promitor side, so I don't think timestamps should be the problem. But that is the whole question, I do not understand why these errors appear in Prometheus logs, and why some metrics are dropped and others are not.

[tomkerkhove]

Unfortunately that's hard for me to understand.🤷‍♂️ You might want to ask the Prometheus community what the best way is to diagnose this, or enable metric timestamps