UDP buffer size problems (FreeBSD, maybe generally when OS != Linux)

[borjam]

Expected Behavior

When setting a buffer size for UDP inputs, either on the Graylog config file (udp_recvbuffer_sizes parameter), the value should be provided to the setsockopt() call. The operating system should decide whether the value is legal or not.

This should work for a value greater than 65536 bytes.

Current Behavior

There are two different issues at play here:

1. At least on FreeBSD there seems to be some buffer size cap to 16 bit (65536) which should not exist.
2. Due to a known linuxism there seems to be some confusion with how to check whether the setsockopt() call has been successful.

The behavior on FreeBSD is rather confusing. Moreover I have found a workaround that suggests a potential fix for any operating system.

On a FreeBSD system configured with a UDP buffer size of 1048576 bytes (1 MB), Graylog gives the following warning:

2024-06-14 12:05:08,300 WARN o.g.i.t.UdpTransport [netty-transport-0] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Alertas DNS ML, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=XXXXXX} (channel [id: 0x75163fc7, L:/10.0.0.1:555]) should be >= 262144 but is 65536.

And the UDP buffer size is indeed 65536 bytes, checking with netstat -x -p udp (not sure these options work on Linux)

This happens in any of these cases:

• udp_recvbuffer_sizes is commented out. The built in default on Graylog seems to be 262144 bytes.
• udp_recvbuffer_sizes is present, specifying a buffer size greater than 65536 bytes. Of course the complaint is "should be >= specified bytes but is 65536")
• Regardless of what happens with the graylog.conf option, specifying a value greater than 65536 bytes results in the same error.

Possible Solution

Workaround on FreeBSD:

• Set the graylog.conf variable udp_recvbuffer_sizes to 0.
• Make sure not to set the optional buffer size on UDP input definitions.

When doing this, Graylog makes a setsockopt(SO_RCVBUF) with 0 specified as a buffer length. The call will fail, so the actual buffer size set by the system to the default value will not change. Graylog issues a warning but it moves forward.

2024-06-15 08:32:31,391 WARN i.n.b.Bootstrap [inputs-0] Failed to set channel option 'SO_RCVBUF' with value '0' for channel '[id: 0x30026dd0]'

Suggested fix:

There is a cap to 65536 bytes somewhere. It may be netty, Graylog or OpenJDK. I don't know.

Anyway in my opinion the sane, foolproof option is to change the default behavior:

• If udp_recvbuffer_sizes is not specified -> Do not try a setsockopt(SO_RCVBUF) unless a specific buffer size is specified for a UDP input of course.
• Alternatively, if udp_recvbuffer_sizes is 0 then do not issue the setsockopt() call that would fail anyway and document this behavior.

Doing this, Graylog will trust whatever default UDP buffer size has been configured by the system administrator which is probably the best option. I am not sure about other systems but at least on FreeBSD there are two system variables controlling this:

• net.inet.udp.recvspace: UDP receive buffer size
• kern.ipc.maxsockbuf: Maximum socket buffer size

Steps to Reproduce (for bugs)

1. On FreeBSD, leave udp_recvbuffer_sizes commented out or specify a value greater than 65536.

Context

This issue is important because the UDP buffer size is critical for high volume UDP packet reception.

Your Environment

• Graylog Version: 6.0.1
• Java Version: OpenJDK BSD Porting Team 17.0.11 on FreeBSD 14.1-RELEASE
• OpenSearch Version: 2.0.14
• MongoDB Version: 5.0.26
• Operating System: FreeBSD 14.1 (amd64)

[drewmiranda-gl]

I was just researching this last week. What i found is that if you're able to configure net.core.rmem_max you can make the error go away:

# The maximum socket receive buffer size which may be set by using the SO_RCVBUF socket option
# Applies immediately, does not persist after a reboot
sudo sysctl -w net.core.rmem_max=1048576
# persists on reboot, but does not apply immediately
echo 'net.core.rmem_max=1048576' | sudo tee -a /etc/sysctl.conf

I validated the above resolves this.

This may be more of a documentation issue, what do you think?

[borjam]

Sorry, different issues.

You are talking about Linux. I am talking about FreeBSD. Not sure about other systems because the problem might be on the netty library or the JDK itself.

So, what happened to me on FreeBSD is this:

1. systl -w net.inet.udp.recvspace=1048576 # This is the right way on FreeBSD
2. Start Graylog with default configuration (it will try to do a setsockopt(SO_RCVBUF) of 262144 bytes.
3. If I check the listening UDP sockets with netstat -x which, on FreeBSD, gives me the actual buffer size, I find out it has ben capped to 65536 bytes.

Either setting the graylog.conf variable or the input buffer size for a given input to a value greater than 65536 bytes results in the buffer size being actually capped.

I confirmed that it is not a problem in my system writing a silly C program. I will attach it below.

Suspecting some kind of buffer size capping I changed the UDP buffer size parameter to 0 on graylog.conf. It is an illegal value of course. But the call fails, as it should, and Graylog just logs a warning and goes on.

The result of the workaround: The buffer size is the one I specified using sysctl net.inet.udp.recvspace.

Now, a small clarification. Not intending to be disrespectful at all, but some people are not aware that FreeBSD is not a Linux distribution. It's an entirely different operating system. Linux also behaves in a really odd way because when you set a buffer size with setsockopt() and you read it back with getsockopt() it returns the specified buffer size multiplied by two.

So, what I think would make all of this work better is:

1. If udp_recvbuffer_sizes is omitted from graylog.conf it would be much better to leave it alone and trust the default values offered by the operating system.
2. When checking the proper buffer size you should take into account that Linux is behaving oddly. Nowadays getsockopt(SO_RCVBUF) returns 2 * the specified buffer size, tomorrow it might be π * the specified buffer size. Anyway getsockopt(SO_RCVBUF) >= specified_value looks reasonable to me.

It would be nice to document the issue anyway. There should be a way to ensure that Graylog does not mess with the buffer size and trusts whatever size has been configured by the system aministrator. It can work with the workaround I found (udp_recvbuffer_sizes=0) which is an ugly kludge anyway, or it can be done better by changing Graylog's behavior when the udp_recvbuffer_sizes variable is omitted.

Now I am running Graylog with udp_recvbuffer_sizes=0 and making sure I don't specify any buffer size for the UDP inputs and it shows the buffer size I wanted (1 MB).

That 65536 byte limit is worth investigating, anyway. Maybe it happens on other platforms.

Checking program: checkbuf.c

/*
 * checkbuf.c
 * Verifies UDP buffer size
 */

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <sys/types.h>
#include <sys/socket.h>


int main(int argc, char **argv)
{
    int s, r;
    unsigned int bufsiz = 0, actual_bufsiz = 0, len = 0;
    
    if ( argc > 1 )
        bufsiz = atoi(argv[1]);
    s = socket(PF_INET, SOCK_DGRAM, 0);
    if ( s < 0 ) {
        perror("socket()");
        exit(1);
    }
    if ( bufsiz > 0 ) {
        r = setsockopt(s, SOL_SOCKET, SO_RCVBUF, &bufsiz, sizeof(bufsiz));
        if ( r < 0 ) {
            perror("setsockopt()");
            exit(1);
        }
    }
    /* Now verify */
    len = sizeof(actual_bufsiz);
    r = getsockopt(s, SOL_SOCKET, SO_RCVBUF, &actual_bufsiz, &len);
    if ( r < 0 ) {
        perror("getsockopt()");
        exit(1);
    }
    if ( bufsiz > 0 )
        printf("SO_RCVBUF: requested: %d, actual: %d\n", bufsiz, actual_bufsiz);
    else
        printf("SO_RCVBUF: %d\n", actual_bufsiz);
    exit(0);
}

[drewmiranda-gl]

My apologies, you did say BSD :)

Also for reference, noting the original discussion via the community forum https://community.graylog.org/t/udp-buffer-size-problem-despite-system-configuration/32726

[borjam]

Just in case it was a JDK bug I "checked" the netty source code and I don't find anything suspect of capping the value to 16 bit. Relevant variables in the associated C code are defined as "int" which should be 32 bit.

And looking at the OpenJDK 17 source code, /usr/local/openjdk17/include/freebsd/jni_md.h also defines "jint" as "int". As it should be.

[borjam]

At least the workaround is working until someone decides to abort if a setsockopt() with an illegal value returns an error :)

Thanks for your attention!
👍