Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uptime-Kuma causing ping floods #4837

Open
2 tasks done
MAMedici opened this issue Jun 7, 2024 · 6 comments
Open
2 tasks done

Uptime-Kuma causing ping floods #4837

MAMedici opened this issue Jun 7, 2024 · 6 comments
Labels
area:monitor Everything related to monitors help question Further information is requested

Comments

@MAMedici
Copy link

MAMedici commented Jun 7, 2024

⚠️ Please verify that this question has NOT been raised before.

  • I checked and didn't find similar issue

🛡️ Security Policy

📝 Describe your problem

I've had problems with uptime-kuma triggering IPS blocks due to excessive/abusive traffic. This latest incident I was able to talk to a network engineer who sent me the logs, Looking at the logs I see my IP is pinging them almost every second, instead of 300s as configured in uptime-kuma for this host.

I ran a packet capture on my firewall and indeed see that uptime-kuma is flooding several hosts with unexpectedly-high ICMP traffic.

I double-checked my site and general configuration settings, and everything loose as it should. Most of my entries are set for 5m (300s).

Any idea what might be causing this and how to make it act as expected? This is Uptime-Kuma 1.23.13 on docker 26.10 on a Debian 12 container on Proxmox. Nothing noteworthy appears in the Uptime-Kuma logs. Firewall logs and capture attached.

📝 Error Message(s) or Log

uptime-kuma ping capture.txt

🐻 Uptime-Kuma Version

1.23.13

💻 Operating System and Arch

Debian 12/Linux 6.5.13-5-pve/amd64

🌐 Browser

Firefox 126.0.1

🖥️ Deployment Environment

Runtime: Docker 26.10
Host: lxc Virtualization running Debian GNU/Linux 12 (bookworm) Linux 6.5.13-5-pve
Architecture: x86-64
Database: sqlite
Filesystem used to store the database on:
number of monitors:

@MAMedici MAMedici added the help label Jun 7, 2024
@CommanderStorm
Copy link
Collaborator

Cannot read the capture you send. Does somewhat look like the wrong interface was being captured, as having proxmox usually means running on a server and those usually don't run via wifi..

image

Have also tried looking at it in a text editor, but that looks like random data to me without an explaination

  • what I would be looking at (what are the columns? What does the formatting shift indicate?)
  • or where said data is coming from.

How many nodes do you have pinging away at said hosting provider with the firewall?

Btw: 1 ping/s = 1 packet/s I don't know why your provider considers this as flooding of multiple hosts. If I were a Provider, I would not worry about that.

@CommanderStorm
Copy link
Collaborator

Also could you link to the docs of your IPS?

I am assuming they don't outright block ICMP at 1 ping/s as

  • that would be way below what would be needed (>= your avaliable bandwith) for a DDoS attack.
  • If said limit wants to prevent network discoverability, you should lower it to 0/s, but I would not recommend this if you want to still have Timeouts, Fragmentation, TTL or Service not Avaliable

@CommanderStorm
Copy link
Collaborator

I have done a bit of digging. Here are the statistics from your capture assuming the second colllumn is the time in µs since the start.
delta s = time since the last ping of the same ip1 and ip2

image

Without additional context what you are actually doing, I don't know if anything is misbehaving

@CommanderStorm CommanderStorm added question Further information is requested area:monitor Everything related to monitors labels Jun 8, 2024
@MAMedici
Copy link
Author

MAMedici commented Jun 8, 2024

Cannot read the capture you send. Does somewhat look like the wrong interface was being captured, as having proxmox usually means running on a server and those usually don't run via wifi..

Sorry, that was a text file that included the logs sent to me from the hosting provider's IPS (top section), and copy-paste from Wireshark of the pcap from my firewall. I've enclosed the pcap in the attached zip file.

How many nodes do you have pinging away at said hosting provider with the firewall?

I am pinging from one node on my end to one IP at the hosting provider.

Btw: 1 ping/s = 1 packet/s I don't know why your provider considers this as flooding of multiple hosts. If I were a Provider, I would not worry about that.

Not flooding multiple hosts at the hosting provider. But looking at the pcap, I can see Uptime Kuma is sending multiple pings several times to multiple hosts, at a rate of around 7-8/s. My expectation (and desire) is that Uptime Kuma would send one ping per defined testing interval for each monitored host/ip.

packetcapture-igb0-20240607155040.zip

@MAMedici
Copy link
Author

MAMedici commented Jun 8, 2024

Without additional context what you are actually doing, I don't know if anything is misbehaving

Attached are the settings for 104.225.208.3

firefox_bVuuSqsjJw

@CommanderStorm
Copy link
Collaborator

CommanderStorm commented Jun 8, 2024

104.225.208.3

Assuming you mean 104.225.208.23.
That monitor had 1 ping in the capture noted above from 74.XX.XX.XX. In the timeframe, of 190s, that is reasonable.
See the diagram in #4837 (comment)

There are a few cases in which we send a retry:

  • Can your host be resolved in all cases?
    If the host cannot be resolved via ipv4, we try again with ipv6 as our library does not provide an error message for this.
  • Have you configured retries? If yes, how many and how often? How often are your moniors down?

In the 189 seconds of traffic that you have approximately the amount of pings that I would expect.

Could you retry this in a less noisy environment and report the results? (docker run ..., then pcap and report the results)
It is currently quite hard to guage if there is a problem or if this is just expected behaviour from my end..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:monitor Everything related to monitors help question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants