Make std fully pluggable

[jethrogb]

@Ericson2314 do you want to write the text for this one?

---

@Ericson2314 taking a stab at it per @jethrogb's request:

Much the various abstractions in std require different implementations on different platforms. Currently, they are implemented for linux, windows, and macos in situ. The advantage of this is dispatch is fully static (done at compile time). The disadvantages of this is

1. It is not modular: it is often necessary to fork std to implement an additional platform, which ties one to a specific version of Rust with the costs of rebases. Even though the Rust team is willing to upstream ports of std to obscure platforms, avoiding the rebase penalty, that work-around will never be totally sufficient because many of us want to develop the platform and the std port simultaneously (e.g. std userland over kernel, or std directly against os-as-a-library).

2. The interface is not checked: There is no automatic verification that the implementations present the same interface (at least the portion we wish to be the same).

As it turns out, these goals are fairly harmonious. As we better define the platform-specific "input interfaces" which are abstracted into std's "output interface", we get both the intended interface for platform-specific crate(s) that can live outside of std, and a mechanism for enforcing that the std interfaces are maintained.

The problem is a a fully perfect solution solution probably requires a lot of language work. The https://github.com/rust-lang/rfcs/blob/master/text/1868-portability-lint.md is needed when different parts of the same crate have different portability requirements, which at least will be the case with std itself but possibly also various platform-specific infra. [Imagine a general "Unix filesystem" implementation which is augmented with linux- or macos-specific functionality.] Also, if we need to "weave" together platform-specific and platform-agnostic functionality in increasingly complex (higher-order, in particular) ways, we will need a more expressive module system like ML's (read about "ML functors" if you are unfamiliar with this.)

Historically, our biggest problem in solving this are that

• library refactors bit-rot quickly, yet require decent buy-in
• language features require even more buy-in
• difficulty in incremental progress makes it hard to build up momentum needed for more ambitious changes

---

Taking a step back, requirements are

• No dynamic dispatch. That rules out trait objects.
• No complicating user-facing types (even if back-compat is preserved). That rules out traits without functions wrapping their methods.
• Platform-specific types, not just functions (e.g. file descriptors). That rules low-level linking tricks on their own.
• No platform-specific panicking on missing behavior. If it builds, it should work.

[Ericson2314]

Speaking personally, now, if I squint, I see both ML-style modules and the portability lint as ways to make coarser-grained modules work better. For example, if every create has the same portability on every single item, one can sort of look at the dependency closure of a package to sort of see what the portability is. Likewise it's sometimes possible to avoid needing ML functors by a proliferation of helper creates (c.f inlining higher-order functions, like the function argument to map).

This is perhaps the primary reason why I advocate for keeping std a facade and staying the "many little crates" course. If we can make good incremental progress splitting off crates like alloc and making them fully pluggable or fully portable, we can build up the momentum needed to push through these more advanced changes. Also, I feel that if there is a (not too ugly) solution to some portability problem with just plain crates v.s. those big guns, I think by virtue of its simplicity and isolation of a small independent interface, it is the better solution. So even if we had all the language features we want, I'd still advocate that route, and thus don't it will just be temporary stop-gap work that needs to be eventually done in practice.