Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolver stops working with Docker 26.1.3 #3274

Open
jeehoonkang opened this issue May 25, 2024 · 5 comments
Open

Resolver stops working with Docker 26.1.3 #3274

jeehoonkang opened this issue May 25, 2024 · 5 comments
Labels
type/bug Bug. Not working as intended

Comments

@jeehoonkang
Copy link

jeehoonkang commented May 25, 2024 •
edited
Loading

Environment & Version

  • Docker: 26.1.3, build b72abbb
  • Docker Compose: v2.27.0
  • Mailu: 2.0.43
  • Platform: Ubuntu 22.04.4

Description

The resolver container stops working after upgrading Docker. (It worked fine.) The symptom is similar to this issue: #2868 Though docker compose exec front ping 1.1.1.1 worked for me (but not docker compose exec front ping google.com).

I temporarily fixed the issue by changing DNS from 172.31.203.254 to 1.1.1.1 for all containers in docker-compose.yml, but I wish to use mailu's resolver again.

Replication Steps

Download a setup from setup.mailu.io and run docker compose up -d.

Observed behaviour

The admin container keeps reporting DNS resolution error, eventually transitioning from "starting" to "unhealthy" status. As a result, the web UI doesn't work.

Expected behaviour

The admin container should succeed in resolving DNS from the resolver container without errors.

Logs

$ docker compose logs admin
[...]
admin-1  | CRITICAL:root:Your DNS resolver at 127.0.0.11 is not working (All nameservers failed to answer the query example.org. IN A: Server 1
27.0.0.11 UDP port 53 answered SERVFAIL). Please see https://mailu.io/master/faq.html#the-admin-container-won-t-start-and-its-log-says-critical
-your-dns-resolver-isn-t-doing-dnssec-validation
[...repeated...]

$ docker exec -it resolver nslookup example.org localhost
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
Server:         localhost
Address:        127.0.0.1#53

** server can't find example.org: SERVFAIL
@nextgens
Copy link
Contributor

What's the output of docker compose logs resolver?

@jeehoonkang
Copy link
Author

@nextgens it's similar with #2868:

[1687623007] unbound[1:0] notice: init module 0: validator
[1687623007] unbound[1:0] notice: init module 1: iterator
[1687623007] unbound[1:0] info: start of service (unbound 1.17.1).

@nextgens
Copy link
Contributor

Are you using netplan? Do you have working internet connectivity in resolver?

What's the output of host example.com 192.168.203.254 ?

@glitterlip
Copy link

I am having the same issue here.
Use unbound without netplan.

It works fine until I decided to change the domain. I tried destroy containers and reconfig and rebuild still the same error in admin container

admin container log

admin-1  | CRITICAL:root:Your DNS resolver at 127.0.0.11 is not working (The resolution lifetime expired after 11.601 seconds: Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.). Please see https://mailu.io/master/faq.html#the-admin-container-won-t-start-and-its-log-says-critical-your-dns-resolver-isn-t-doing-dnssec-validation

host resover

resolvectl status | grep DNSSEC
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
          DNSSEC NTA: 10.in-addr.arpa
      DNSSEC setting: allow-downgrade

logs in front container exec ping

docker compose exec front ping example.com
WARN[0000] /root/mailu/docker-compose.yml: `version` is obsolete
PING example.com (93.184.215.14): 56 data bytes
64 bytes from 93.184.215.14: seq=0 ttl=51 time=159.437 ms
64 bytes from 93.184.215.14: seq=1 ttl=51 time=159.393 ms
64 bytes from 93.184.215.14: seq=2 ttl=51 time=159.330 ms
64 bytes from 93.184.215.14: seq=3 ttl=51 time=159.168 ms
64 bytes from 93.184.215.14: seq=4 ttl=51 time=159.106 ms

exec host command in resolver container

host example.com 192.168.203.254
Using domain server:
Name: 192.168.203.254
Address: 192.168.203.254#53
Aliases:

example.com has address 93.184.215.14
example.com has IPv6 address 2606:2800:21f:cb07:6820:80da:af6b:8b2c
example.com mail is handled by 0 .

logs in resolver

resolver-1  | [1717317750] unbound[1:0] notice: init module 0: validator
resolver-1  | [1717317750] unbound[1:0] notice: init module 1: iterator
resolver-1  | [1717317750] unbound[1:0] info: start of service (unbound 1.20.0).
resolver-1  | [1717317751] unbound[1:0] info: generate keytag query _ta-4f66. NULL IN
resolver-1  | [1717319047] unbound[1:0] info: service stopped (unbound 1.20.0).
resolver-1  | [1717319047] unbound[1:0] info: server stats for thread 0: 410 queries, 101 answers from cache, 309 recursions, 0 prefetch, 0 rejected by ip ratelimiting
resolver-1  | [1717319047] unbound[1:0] info: server stats for thread 0: requestlist max 7 avg 2.64725 exceeded 198 jostled 0
resolver-1  | [1717319047] unbound[1:0] info: average recursion processing time 0.661231 sec
resolver-1   | [1717319047] unbound[1:0] info: histogram of recursion processing times

currently /admin,/webmail returns 502

in front container

curl http://localhost/webmail/ -i
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Sun, 02 Jun 2024 09:46:11 GMT
Content-Type: text/html
Content-Length: 170
Connection: close

<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>

log shows connection refused

front-1  | 2024/06/02 09:46:11 [error] 17#17: *48 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /webmail/ HTTP/1.1", subrequest: "/internal/auth/user", upstream: "http://192.168.203.5:8080/internal/auth/user", host: "localhost"
front-1  | 2024/06/02 09:46:11 [error] 17#17: *48 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /webmail/ HTTP/1.1", host: "localhost"
front-1  | 127.0.0.1 - - [02/Jun/2024:09:46:11 +0000] "GET /webmail/ HTTP/1.1" 500 170 "-" "curl/8.5.0"

here are containers

docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED             STATUS                      PORTS                                                                                                                                                                                                                                                                           NAMES
2d6ca5d92d91   ghcr.io/mailu/fetchmail:2.0   "/fetchmail.py"          About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-fetchmail-1
7dfeac40b955   ghcr.io/mailu/webmail:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-webmail-1
7e351fa47236   ghcr.io/mailu/dovecot:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)     110/tcp, 143/tcp, 993/tcp, 2525/tcp, 4190/tcp                                                                                                                                                                                                                                   mailu-imap-1
e21a60b12f58   ghcr.io/mailu/postfix:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-smtp-1
1fce0b8aa877   ghcr.io/mailu/rspamd:2.0      "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (unhealthy)                                                                                                                                                                                                                                                                                   mailu-antispam-1
be501512694e   ghcr.io/mailu/admin:2.0       "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (unhealthy)                                                                                                                                                                                                                                                                                   mailu-admin-1
25b440111eb2   ghcr.io/mailu/nginx:2.0       "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)     156.238.240.200:25->25/tcp, 156.238.240.200:110->110/tcp, 156.238.240.200:143->143/tcp, 156.238.240.200:465->465/tcp, 156.238.240.200:587->587/tcp, 156.238.240.200:993->993/tcp, 156.238.240.200:995->995/tcp, 156.238.240.200:10080->80/tcp, 156.238.240.200:10443->443/tcp   mailu-front-1
711184d8dc97   redis:alpine                  "docker-entrypoint.s…"   About an hour ago   Up 17 minutes               6379/tcp                                                                                                                                                                                                                                                                        mailu-redis-1
bdc97ffbbb23   ghcr.io/mailu/oletools:2.0    "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-oletools-1
f11df031a910   ghcr.io/mailu/unbound:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-resolver-1

f11 is resolver

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' f11
192.168.203.254

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' be
192.168.203.5

@glitterlip
Copy link

I am having the same issue here. Use unbound without netplan.

It works fine until I decided to change the domain. I tried destroy containers and reconfig and rebuild still the same error in admin container

admin container log

admin-1  | CRITICAL:root:Your DNS resolver at 127.0.0.11 is not working (The resolution lifetime expired after 11.601 seconds: Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.; Server 127.0.0.11 UDP port 53 answered The DNS operation timed out.). Please see https://mailu.io/master/faq.html#the-admin-container-won-t-start-and-its-log-says-critical-your-dns-resolver-isn-t-doing-dnssec-validation

host resover

resolvectl status | grep DNSSEC
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
          DNSSEC NTA: 10.in-addr.arpa
      DNSSEC setting: allow-downgrade

logs in front container exec ping

docker compose exec front ping example.com
WARN[0000] /root/mailu/docker-compose.yml: `version` is obsolete
PING example.com (93.184.215.14): 56 data bytes
64 bytes from 93.184.215.14: seq=0 ttl=51 time=159.437 ms
64 bytes from 93.184.215.14: seq=1 ttl=51 time=159.393 ms
64 bytes from 93.184.215.14: seq=2 ttl=51 time=159.330 ms
64 bytes from 93.184.215.14: seq=3 ttl=51 time=159.168 ms
64 bytes from 93.184.215.14: seq=4 ttl=51 time=159.106 ms

exec host command in resolver container

host example.com 192.168.203.254
Using domain server:
Name: 192.168.203.254
Address: 192.168.203.254#53
Aliases:

example.com has address 93.184.215.14
example.com has IPv6 address 2606:2800:21f:cb07:6820:80da:af6b:8b2c
example.com mail is handled by 0 .

logs in resolver

resolver-1  | [1717317750] unbound[1:0] notice: init module 0: validator
resolver-1  | [1717317750] unbound[1:0] notice: init module 1: iterator
resolver-1  | [1717317750] unbound[1:0] info: start of service (unbound 1.20.0).
resolver-1  | [1717317751] unbound[1:0] info: generate keytag query _ta-4f66. NULL IN
resolver-1  | [1717319047] unbound[1:0] info: service stopped (unbound 1.20.0).
resolver-1  | [1717319047] unbound[1:0] info: server stats for thread 0: 410 queries, 101 answers from cache, 309 recursions, 0 prefetch, 0 rejected by ip ratelimiting
resolver-1  | [1717319047] unbound[1:0] info: server stats for thread 0: requestlist max 7 avg 2.64725 exceeded 198 jostled 0
resolver-1  | [1717319047] unbound[1:0] info: average recursion processing time 0.661231 sec
resolver-1   | [1717319047] unbound[1:0] info: histogram of recursion processing times

currently /admin,/webmail returns 502

in front container

curl http://localhost/webmail/ -i
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Sun, 02 Jun 2024 09:46:11 GMT
Content-Type: text/html
Content-Length: 170
Connection: close

<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>

log shows connection refused

front-1  | 2024/06/02 09:46:11 [error] 17#17: *48 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /webmail/ HTTP/1.1", subrequest: "/internal/auth/user", upstream: "http://192.168.203.5:8080/internal/auth/user", host: "localhost"
front-1  | 2024/06/02 09:46:11 [error] 17#17: *48 auth request unexpected status: 502 while sending to client, client: 127.0.0.1, server: , request: "GET /webmail/ HTTP/1.1", host: "localhost"
front-1  | 127.0.0.1 - - [02/Jun/2024:09:46:11 +0000] "GET /webmail/ HTTP/1.1" 500 170 "-" "curl/8.5.0"

here are containers

docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED             STATUS                      PORTS                                                                                                                                                                                                                                                                           NAMES
2d6ca5d92d91   ghcr.io/mailu/fetchmail:2.0   "/fetchmail.py"          About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-fetchmail-1
7dfeac40b955   ghcr.io/mailu/webmail:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-webmail-1
7e351fa47236   ghcr.io/mailu/dovecot:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)     110/tcp, 143/tcp, 993/tcp, 2525/tcp, 4190/tcp                                                                                                                                                                                                                                   mailu-imap-1
e21a60b12f58   ghcr.io/mailu/postfix:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-smtp-1
1fce0b8aa877   ghcr.io/mailu/rspamd:2.0      "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (unhealthy)                                                                                                                                                                                                                                                                                   mailu-antispam-1
be501512694e   ghcr.io/mailu/admin:2.0       "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (unhealthy)                                                                                                                                                                                                                                                                                   mailu-admin-1
25b440111eb2   ghcr.io/mailu/nginx:2.0       "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)     156.238.240.200:25->25/tcp, 156.238.240.200:110->110/tcp, 156.238.240.200:143->143/tcp, 156.238.240.200:465->465/tcp, 156.238.240.200:587->587/tcp, 156.238.240.200:993->993/tcp, 156.238.240.200:995->995/tcp, 156.238.240.200:10080->80/tcp, 156.238.240.200:10443->443/tcp   mailu-front-1
711184d8dc97   redis:alpine                  "docker-entrypoint.s…"   About an hour ago   Up 17 minutes               6379/tcp                                                                                                                                                                                                                                                                        mailu-redis-1
bdc97ffbbb23   ghcr.io/mailu/oletools:2.0    "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-oletools-1
f11df031a910   ghcr.io/mailu/unbound:2.0     "/bin/sh -c /start.py"   About an hour ago   Up 17 minutes (healthy)                                                                                                                                                                                                                                                                                     mailu-resolver-1

f11 is resolver

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' f11
192.168.203.254

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' be
192.168.203.5

changing 192.168.203.254 to 1.1.1.1 fixed the problem

@nextgens nextgens added the type/bug Bug. Not working as intended label Jun 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Bug. Not working as intended
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nextgens @jeehoonkang @glitterlip