pageserver: direct I/O for reads

[jcsp]

Currently, we do buffered reads of data pages. Direct reads would be a better fit, because:

• Pageserver data pages have an extremely low temporal locality on reads, because any repeatedly accessed pages are cached inside postgres. This makes it largely a waste of memory, which we could be using for other things.
• The kernel page cache gives deceptively fast read performance on lightly loaded pageservers, making performance less consistent as pageservers are packed with larger numbers of tenants.

Tasks

Beta Give feedback

1. predicting/testing perf impact of direct IO (inject delays into reads that currently hit RAM?) -- test the assumption that disks are fast enough and our bottleneck is mostly software.
2. changing code base to use 4k-aligned-buffers everywhere, and validate this by having VirtualFile panic!() if it sees an unaligned buffer
3. feature-flagged switch to issue IOs with O_DIRECT
4. Improve userspace PageCache monitoring, where we sort index pages & expect to have a high hit rate
5. Reap the benefits of no longer using so much RAM for kernel page cache: make PageCache (and any other userspace caches we choose to add) larger.

Backpointer to the Slack DMs between John and Christian about this: https://neondb.slack.com/archives/D05KTCVS40H/p1718977335312439