Microkernels

If you want a complete description of what microkernels are then read the wikipedia article, but in one sentence it's a kernel where everything is in userspace except for scheduling, managing access to I/O devices, and IPC.

Why microkernels?

Better security, better reliability, better modularity. Better security because in a muicrokernel system a bug in one driver only gives an attacker access to that subsystem (or maybe even that driver), as opposed to having unlimited access to the whole system like is the case with mainstream operating systems today. Better relaibility becasue a crash in one subsystem only crashes that subsystem instead of crashing everything; if windows were a microkernel system then the crowdstrike bug would have just stopped some IT security staff from getting telemetry instead of being front page news. If linux were a microkernel system then the linux kernel team wouldn't be responsible for merging every driver for every hardware device, and wouldn't be responsible for vetting code that they would have to become experts on the inner workings of every chip supported by the linux kernel to properly vet.

Why not microkernels?

If microkernels are so much better why isn't every OS a microkernel OS? Lots of people were asking that question in the 80s and 90s, and a lot of them made microkernel OS's, and it turned out the answer was that the overhead was too high.¹

The overhead was too high because computers in the 80s didn't have a way for a userspace process to directly access a particular hardware device. That means that every time eg: the filesystem wanted to do a disk read, it had to do a system call, which triggered a context switch, which triggered a whole bunch of expensive locking and memory copying.

Why microkernels now?

In a word: IOMMU. Virtually all PCs in use today have IOMMUs, they've been standard issue for about a decade. By using IOMMU and shared memory in the right way you can completely eliminate context switches from the happy path (which is the case where you have enough cores that all the work that needs to be done can be done by processes that are currently running), and virtually elimiate it overall if you're willing to accept a little bit of latency.

There are three key things that the kernel has to do in a microkernel system: scheduling, inter-process communication, and managing access to I/O devices.

The scheduler for a microkernel that's based on virtualization technology (which IOMMMU is part of) would look like the Xen hypervisor.

Access to I/O devices is managed in hardware by the IOMMU.

For zero-overhead (at least in the happy path) IPC all you need is the ability to allocate shared buffers between processes and integer atomic compare-and-swap. That lets you use the shared buffer as a command queue; just a ring buffer with atomics on bumping the start and end pointers. That lets you asyncrhonously pass messages with no context switches, no copying between address spaces and no locks. This pattern is used a lot in things like GPU drivers today.

What about shared libraries?

How do shared libraries work when your OS processes are actually VM guests? Link them into the program at launch time and implement everything that doesn't need to happen in another process locally, exokernel stlye. These days every app ships with a copy of its own operating system anyway (in the form of electron), so having redundant copies of libraries in memory isn't the issue it was thirty years ago.

What would it take to actually implement something like this?

Actually not that much. Xen already has most of what you need for the hypervisor layer. You can copy what Mach did and implement the servers for network, filesystem, etc by copying the code from freebsd. DRM is already based around asynchronous command buffers so you could copy the graphics subsystem from linux and run it in userspace (though it might be convenient to run the display server in the same process).