a week ago, I got a very interesting idea that I'll explain and share here about windows environment subsystems
first, I want to explain 3 different terms that I'll use:
- environment subsystem:
an environment subsystem is responsible of exposing a subset of the native executive functionalities in a clean API called "subsystem DLLs", and it typically consists of a kernel-mode part, which is a driver that managed kernel-mode operations required for the environment subsystem to function, user-mode part, which is in most cases a process that takes notifications for different events (process creation/termination, threads creation/termination, etc..) that relates to this specific subsystem and subsystem DLLs which are the API of the environment subsystem that an application links against to use its functionality.
- application subsystem:
an environment subsystem can include more than application subsystem, for example, Windows (win32) environment subsystem includes GUI application subsystem, which manages GUI apps and CUI application subsystem (the console subsystem) which manages CUI applications.
- kernel/executive subsystem:
this is a part of the kernel or the executive layer that manages a specific layer of NT kernel, for example, the object manager subsystem which manages all the objects in the system, and DirectX graphics subsystem which manages DirectX technology.
historically speaking, NT kernel was designed to support multiple environment subsystems: OS/2, POSIX and win32, now only win32 exists and no support for other subsystems is officially maintained by Microsoft
my goal is to replace win32 environment subsystem (and as a consequence, its application subsystems) with a custom one, giving infinite freedom to customize windows while taking benefit of the extensive HAL (hardware abstraction layer) and hardware acceleration that Microsoft provides!
but this isn't easy nor I'm sure it's even possible, it requires syncing different parts of windows and communication with different applications... I'll call my theoretical environment subsystem "nexus", so let's talk about what should nexus do to get loaded as an environment subsystem instead of win32:
1- it should handshake with SMSS using ALPC IPC mechanism:
SMSS (session manager subsystem) is the first user-mode process created, and it's responsible of making the initial sessions and loading the environment subsystem registered in the registry under this key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems
specifically, Required and kmode keys, Required field contains the major environment subsystem that should be loaded upon boot, and kmode contains the kernel mode part of it, there is another field called optional that contains the optional environment subsystem that'll be loaded on-demand, and it was used with POSIX environment subsystem
SMSS uses ALPC IPC mechanism (refer to
this article and
this lecture for more information about ALPC), any environment subsystem should interact with SMSS in a specific, undocumented message structure and order, nexus should replicate the communication between csrss.exe (the win32 environment subsystem user-mode process) and smss.exe
2- it should manage different hardware components:
NT kernel and its executive layer and different drivers are kernel-mode components, an environment subsystem should wrap all these messy components in a unified, clean API, win32 environment subsystem does this using win32k* drivers which manage the communication with DirectX graphics kernel driver (dxgkrnl.sys) and mouse and keyboard input, nexus should also make a similar driver, and the biggest hurdle in this layer isn't implementation details as much as its the design itself, because this layer is which defines the characteristics and features of the whole environment subsytem!
3- it should provide environment subsystem DLLs and application subsystems
the first one is easy, it'll be "yet another libc implementation and wrappers" style thingy, but the application subsystems are a major part of an environment subsystems, and designing a solid-yet-flexible application subsystems architecture isn't an easy task!
4- it shouldn't annoy patch guard
update:
I searched a bit about this, it seems like PatchGuard is an ally in this context!
if I understand well, PatchGuard scans the SSDT (and other structures) after loading the environment subsystem and takes its content as a reference, then periodically, and if it found any change, it'll immediately trigger a BSOD... but since the reference scan happens after loading the environment subsystem, I could register syscalls for nexusk driver in the second slot of SSDT, which win32k used to use, and PatchGuard should happily protect it from temper! turning to a world-class protection system for my own environment subsystem!!
disclaimer:
This idea is highly experimental and based on limited research. Many assumptions may be incorrect or incomplete.
However, if all components are handled correctly, it might be possible to:
-
Boot the NT kernel
-
Load a custom environment subsystem
-
Render output via the graphics kernel
-
Run a minimal user environment
Motivation
Why attempt something like this?
1. Curiosity
Understanding Windows internals at a deep level.
2. Performance Potential
In theory, this could produce a system that is:
-
Extremely minimal
-
Possibly faster than Linux (I wish!)
3. Driver Model Advantage
Windows has a strong driver ecosystem, especially for GPUs.
goals:
these are the goals that I should achieve to say that this idea succeeded:
1- be able to load my environment subsystem components (kernel mode, user mode and environment subsystem DLLs) as the only environment subsystem.
2- be able to communicate with VidPN manager and dxgkrnl.sys and wrap them in a unified interface to manager display in the system
3- provide a C standard library implementation (both statically and dynamically linked libraries) to provide basic functionalities for user mode applications to work
3- make a basic TTY environment and shell with some coreutils to demonstrate the environment subsystem is capable of actually working and performing various operations.
sources:
here are some sources that I'm studying and I'd recommend:
1- windows internals 7th editions (part 1 and 2), these are the most valuable for understanding win32 environment subsystem in order to replace its functionalities
2- OSR forums articles, this is the place where you'll find a nerd explaining the exact problem you're facing and giving you the optimal solution for it anonymously
3- react OS project source code (especially their implementation for smss and csrss), while they're still working on WDDM and dxgkrnl and still working on NT 5.2 (windows server 2003 version), their codebase is very important to clarify stuff even reverse engineering can't
4- dxg driver for linux (
link), which provides a good interface to communicate directly with dxgkrnl.sys driver via IOCTLs, this is a definitive source of microsoft itself, no guesses or hacks and totally works with windows 11 and its graphics stack
5- Midipix project: this is a custom musl libc + POSIX API post built up on NT native API to make porting UNIX-targetted source code easier, it's a very rich source for NT native API reverse engineering