smp: add a function that barriers memory prefault work

[tomershafir]

Currently, memory prefault logic is internal and seastar doesnt provide much control to users. In order to improve the situation, I suggest to provide a barrier for the prefault threads. This allows to:

• Prefer predictable low latency and high throughput from the start of request serving, at the cost of a startup delay, depending on machine characteristics and application specific requirements. For example, a fixed capacity on prem db setup, where slower startup can be tolerated. From users perspective, they generally cannot tolerate inconsistency (like spikes in latency).
• Similarly, improve user scheduling decisions, like running less critical tasks while prefault works.
• Reliably test the prefault logic, improving reliability and users trust in seastar.

I tested locally. If you approve, next I will try to submit a prefault test.

[avikivity]

Did you observe latency impact from the prefault threads? It was written carefully not to have latency impact, but it's of course possible that some workloads suffer.

[tomershafir]

As you described in #1702, page faults can cause deviation, and following up the example, there can be 25sec where latency is variably higher.

[avikivity]

As you described in #1702, page faults can cause deviation, and following up the example, there can be 25sec where latency is variably higher.

I said nothing about latency being higher there.

We typically run large machines with a few vcpus not assigned to any shards, and the prefault threads run with low priority.

[tomershafir]

There are 2 aspects:

1. Page faults

In the previous comment, I meant page fault latency. The page faults can cause high latency unpredictably until the prefaulter finishes.

Regarding page faults measurement, it seems I cannot reliably measure on my env.

2. Prefault threads competition

I tried to non scientifically isolate wall time overhead of prefault threads:

I have a test app that performs file I/O and process memory buffers repeatedly. I used Ubuntu Orbstack VM with 1 NUMA node, 10 cores, --memory=14G - effectively a small NUMA node, and a small input to let the overhead be most visible.

• With --lock-memory=1 without waiting, I see that the chrono time of the actual work is significantly higher than with --lock-memory=0. (~1800ms > ~600ms)
• When waiting before doing actual work, I see that the overhead is removed.
• When building seastar without prefault code and --lock-memory=1 I dont see the overhead.

[tomershafir]

By default seastar uses all vcpus, which makes sense for resource efficiency.

Also, do you free specific vcpus? Like one per numa node, the granularity of prefault threads.

[avikivity]

By default seastar uses all vcpus, which makes sense for resource efficiency.

Also, do you free specific vcpus? Like one per numa node, the granularity of prefault threads.

1 in 8, with NUMA awareness. They're allocated for kernel network processing. See perftune.py.

[tomershafir]

Nice. Let me know if this change makes sense to you

[tomershafir]

@avikivity ping

[tomershafir]

I also tried to simulate perftune with 1 free vcpu: --cpuset=0-8 given the above setup, and I still observe the overhead, even though its less (~1600ms).