UDP Response Timeout in Bridge Mode Networking

[daveismith]

Description

I'm trying to run a LWM2M server inside of docker for UDP based clients to connect to. These clients will register with the server using a CoAP message and specify a lifetime for the registration. Periodically they update this registration to maintain communication with the server.

When the server desires information from the client, it will send a CoAP request to the same UDP port as the client registered from. I'm using Eclipse Leshan for this.

When I put the server into a container, I see that there are windows after the registration/update messages where I can't query the client from the server. This doesn't happen if I run the application natively.

In digging down further, I also discovered that this doesn't happen in host networking mode, only when using port mapping via bridge (I've only tested with the default network so far). I've seen this behaviour in bridge networking mode on both docker for mac (apple silicon) and a raspberry pi. This makes me think that this is likely some sort of NAT timeout happening in the bridge network stack. Unfortunately, host mode networking isn't an option on OS X and windows from my understanding so just switching to host mode networking isn't an option.

I wrote a simple pair of python programs to demonstrate the issue, which are included below. They work by receiving a UDP datagram and then waiting for a period of time before echoing it back to the sender. This works in a docker container for delays up to 89 seconds. Once the value increases past this, no response is received or seen in wireshark. Running the app natively I've tested with delays up to 900 seconds and not seen any issues.

Reproduce

Server

# basic echo server with a delay
import socket
import time

UDP_IP = "::"
UDP_PORT = 5005

sock = socket.socket(socket.AF_INET6,  # Internet
                     socket.SOCK_DGRAM)  # UDP
sock.bind((UDP_IP, UDP_PORT))

while True:
    data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
    print("received message: %s" % data)
    #time.sleep(900)     # 15 minutes works perfectly on the local machine outside docker. Also works on a raspberry pi running outside docker or in host network mode
    time.sleep(90)    # this does not work in docker (default bridge)
    # time.sleep(89)      # this works in docker (89 works)
    print("sending %s" % data)
    sock.sendto(data, addr)

Client

# Basic Echo Client
import socket

UDP_IP = "::1"
UDP_PORT = 5005
MESSAGE = b"Hello, World!"

print("UDP target IP: %s" % UDP_IP)
print("UDP target port: %s" % UDP_PORT)
print("message: %s" % MESSAGE)

sock = socket.socket(socket.AF_INET6, # Internet
                     socket.SOCK_DGRAM) # UDP
sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print("received message: %s" % data)

Dockerfile

FROM python:3.9
ADD server.py .

EXPOSE 5005/udp
ENTRYPOINT [ "/usr/local/bin/python3" ]
CMD ["/server.py"]

1. docker build -t udp_test .
2. docker run -p 5005:5005/udp udp_test
3. If running on a separate machine, modify the UDP_IP in the client
4. python3 client.py
5. Observe that while the server indicates it sent data (line 18), the client never receives the data
6. modify the timeout to 89 seconds and re-run step 4
7. Observe that the server indicates it sent data and the client indicates it was received

Expected behavior

The UDP message sent by the server should be received by the client. Ideally this would not require special configuration of the docker container, but if there is a need to adjust a default timeout, this is documented clearly and available for use in docker compose.

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       ac2de55
  Built:            Tue Apr 30 11:48:04 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /Users/davids/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /Users/davids/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /Users/davids/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/davids/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/davids/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/davids/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/davids/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/davids/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /Users/davids/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 5
  Paused: 0
  Stopped: 5
 Images: 72
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.26-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: c3d42315-4347-4246-b570-b032255b9cba
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/davids/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Additional Info

No response

[robmry]

Hi @daveismith - thank you for taking the time to narrow down the problem and provide the example.

I thought I had it ... using Docker Engine (docker-ce) on Linux without Docker Desktop, your repro showed the issue and I found that docker-proxy has a hard-coded 90s timeout. (I changed the timeout, made a local build of docker-proxy, and it had the expected effect. Running without docker-proxy solved the problem.)

Then I tried your repro with Docker Desktop on MacOS, it demonstrates the issue there too - but I now realise Docker Desktop doesn't need to use docker-proxy. So, there must be a variation on the theme ... I'll need to dig deeper / ask around (and maybe we'll need to raise a separate issue, but I'll try to see what's what first).

[daveismith]

Thanks @robmry. I'm glad to hear that for at least the linux side the hypothesis was correct. Hopefully you're able to narrow it down in the MacOS side as well, but a separate ticket would make a lot of sense to me. I haven't tried on Windows as I don't have a windows machine available but I think it's likely the same behaviour exists there.