Matterbridge gets into endless loop of trying to connect to Discord after failure, causing bot token to be revoked

[thomasjwebb]

I don't quite understand what's happening but it appears that occasionally matterbridge will fail to connect to Discord due to a connection issue. Once that happens, the logs get flooded with the same message over and over about failure to send heartbeat to Discord. Then presumably starting from the time the bot token was reset by Discord for all these repeated failed attempts, then it enters an endless loop where it successfully connects to everything except Discord and since that failed, it restarts and tries again. It will never succeed because the bot token was revoked.

So the main issue is that simply failing to connect even once is enough for it to enter a bad state that will inevitably cause me to have to go through the annoying process of getting a new bot token, adding the bot to the channels, etc. And the second issue that it keeps trying even though it will never succeed. I would rather matterbridge just shut down when this happens. But more importantly, it needs to not enter the endless loop that causes the bot token to be revoked.

This is what the log looks like each time this happens:

Jun  5 03:03:25 synapse matterbridge[92386]: 2024/06/05 03:03:25 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, write tcp 10.0.0.2:28996->162.159.136.234:443: write: broken pipe
Jun  5 03:03:32 synapse matterbridge[92386]: 2024/06/05 03:03:32 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:03:37 synapse matterbridge[92386]: 2024/06/05 03:03:37 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:03:50 synapse matterbridge[92386]: 2024/06/05 03:03:49 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:04:13 synapse matterbridge[92386]: 2024/06/05 03:04:13 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:04:41 synapse matterbridge[92386]: 2024/06/05 03:04:41 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:04:44 synapse matterbridge[92386]: 2024/06/05 03:04:44 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:04:53 synapse matterbridge[92386]: 2024/06/05 03:04:53 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:04:59 synapse matterbridge[92386]: 2024/06/05 03:04:59 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:05 synapse matterbridge[92386]: 2024/06/05 03:05:05 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:09 synapse matterbridge[92386]: 2024/06/05 03:05:09 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:15 synapse matterbridge[92386]: 2024/06/05 03:05:15 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:21 synapse matterbridge[92386]: 2024/06/05 03:05:21 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:24 synapse matterbridge[92386]: 2024/06/05 03:05:24 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:30 synapse matterbridge[92386]: 2024/06/05 03:05:30 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:39 synapse matterbridge[92386]: 2024/06/05 03:05:39 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:44 synapse matterbridge[92386]: 2024/06/05 03:05:44 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:05:49 synapse matterbridge[92386]: 2024/06/05 03:05:49 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:06:34 synapse matterbridge[92386]: 2024/06/05 03:06:34 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, write tcp 10.0.0.2:42984->162.159.136.234:443: write: broken pipe
Jun  5 03:06:34 synapse matterbridge[92386]: 2024/06/05 03:06:34 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:06:36 synapse matterbridge[92386]: 2024/06/05 03:06:36 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:06:40 synapse matterbridge[92386]: 2024/06/05 03:06:40 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:07:38 synapse matterbridge[92386]: 2024/06/05 03:07:38 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:07:39 synapse matterbridge[92386]: 2024/06/05 03:07:39 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent
Jun  5 03:07:43 synapse matterbridge[92386]: 2024/06/05 03:07:43 [DG0] wsapi.go:302:heartbeat() error sending heartbeat to gateway wss://gateway.discord.gg/?v=8&encoding=json, websocket: close sent

Then it keeps going with just that last line, "close sent" over and over again 1000s of times. It's very non-deterministic. I've gone months without this happening. Seems like it happens more lately. I wonder if Discord is less reliable. I'm certain there's nothing wrong with my matterbridge server's connection. It just doesn't handle not being able to connect well.

Environment info:

$ uname -a
FreeBSD synapse 13.3-RELEASE-p1 FreeBSD 13.3-RELEASE-p1 GENERIC amd64
$ matterbridge --version
version: 1.26.0

Config file section for discord:

[discord.mfhz]
Token="REDACTED"
Server="REDACTED"
ShowEmbeds=false
UseUserName=false
UseDiscriminator=false
EditDisable=false
EditSuffix=" (edited)"
RemoteNickFormat="[{PROTOCOL}] <{NICK}> "
ShowJoinPart=false
StripNick=false
ShowTopicChange=false
SyncTopic=false