Allow running Driver threads with SCHED_BATCH policy on Linux

[arhimondr]

Description

Use SCHED_BATCH scheduling policy for Driver thread pool

Motivation and Context

With blocking IO used by connectors (such as Hive connector) often it is necessary to set the number of threads to the number higher than the number of available cores.

This is needed to avoid slowdowns for IO heavy workload.

However for CPU intensive queries it may create unnecessary thread contention and may cause stability problems when communication threads are delayed.

Threads scheduled with SCHED_BATCH policy are run with slightly lower priority giving a green light for communication threads. These can run for longer resulting in less cache flushes if only other batch threads are waiting for execution.

More information about different scheduling policies in Linux can be found here: https://man7.org/linux/man-pages/man7/sched.7.html

Impact

Improves efficiency and stability for certain cluster configurations

Test Plan

• Set driver.threads-batch-scheduling-enabled=true
• Find a Driver thread
• Run chrt -p <driver thread id>

Result:

chrt -p 3238430
pid 3238430's current scheduling policy: SCHED_BATCH
pid 3238430's current scheduling priority: 0

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== NO RELEASE NOTE ==

[xiaoxmeng]

@arhimondr thanks % minors.

[xiaoxmeng]

Do we see improvement with a dedicated cpu executor for this?

[arhimondr]

When a Driver executor is busy doing computation it may result in increase in Exchange request latency, slowing down data exchange. The idea is that having a separate executor should decouple Exchange communication from execution potentially improving throughput.

[xiaoxmeng]

Can we log a warning instead of fail the server start if the non-linux platform doesn't this?

[arhimondr]

The property is false by default and shouldn't be set on systems other than Linux. Failing loud when somebody is trying to set it on systems other than Linux seems to less error prone.

[xiaoxmeng]

Have we tuned on this? Thanks!

[arhimondr]

This is the only supported value by SCHED_BATCH. It fails when I set it to any different value.

[arhimondr]

Thanks for the review @xiaoxmeng , updated

[xiaoxmeng]

@arhimondr thanks!