whitemagic/openspec/changes/whitemagic-foundation/design.md

## Context

This repo contains four C# process-manipulation libraries studied in `docs/memory-library-comparison.md`:

- **BlackMagic-old** (`reference/Blackmagic-old/`) — .NET FW 4.0, x86, FASM as a public `Asm` property.
- **MemorySharp** (`reference/MemorySharp/`) — .NET FW, x86, FASM behind `Execute<T>`; rich high-level API (PEB/TEB, window, input, calling conventions) but unmaintained since 2016.
- **GreyMagic** (`reference/GreyMagic/`) — .NET FW, ~2016, x86; dual in/out-of-process `MemoryBase`, `MarshalCache`, `DetourManager`, `PatchManager`, `CreateFunction<T>`, `PeHeaderParser`.
- **BlackMagic** (`reference/Blackmagic/`, current) — .NET 8, x86 + x64, FASM removed, pattern scanning + cache, DLL injection (CreateThread + hijack), hand-assembled x86 stubs (`SInject.cs` `EmitU8`/`EmitU32` byte emitter), `CreateRemoteThread`-based `Execute` (`BMThread.cs`). No frame hook exists.

The consuming use case is an automation client targeting a legacy x86 desktop application. The dominant failure mode observed (`FASM-MIGRATION.md`) is target crashes when state-sensitive functions are invoked on a thread created by `CreateRemoteThread`, because the target's main thread has exclusive affinity for its scripting VM, render device, and object model.

WhiteMagic is a **new, additive** .NET 8 library that unifies the four. It reuses their ideas, not their assemblies. No project already depends on WhiteMagic, so there is no backward-compatibility constraint.

## Goals / Non-Goals

**Goals:**
- Single modern (.NET 8, nullable, `Span<byte>`, `SafeHandle`) library that is bitness-agnostic (x86 + x64).
- A **three-tier execution model** whose default path for state-sensitive calls is crash-safe (runs on the target's own thread), while `CreateRemoteThread` remains available for thread-agnostic payloads.
- Dual memory access: out-of-process (RPM/WPM) and in-process (RPM-on-self-handle, see D1 revision) behind one abstract `MemoryBase`, with `MarshalCache<T>` for allocation-free typed IO.
- Reversible function hooking (`DetourManager`) and byte patching (`PatchManager`) with auto-restore on dispose.
- Replace FASM with an `IAssembler` seam: hand-emitted convention stubs by default, optional Iced backend for arbitrary assembly. Zero native dependency in the default configuration.
- Port DLL injection (CreateThread + thread-hijack, x86/x64) and pattern scanning + cache from current BlackMagic.
- Provide MemorySharp-grade ergonomics: `RemotePointer` indexer, `Module["fn"].Execute(...)`, PEB/TEB, window mutation, input simulation, async wrappers, helpers.
- Test-first for all pure logic (assembler encodings, marshal cache, pattern matching, stub building, pump queue semantics).

**Non-Goals:**
- Modifying or replacing BlackMagic/MemorySharp/GreyMagic — WhiteMagic is additive.
- Shipping a full x86/x64 assembler ourselves — arbitrary assembly is delegated to Iced; only the fixed convention-stub shapes are hand-emitted.
- Managed-DLL injection bootstrapper (the CLR host that loads `InProcessReader` into the target). WhiteMagic exposes the in-process API surface; wiring an actual managed loader is a follow-up change.
- Application-specific offsets, scripting-engine hooks, or automation logic — those live in the consumer, not the library.
- Interference with other software's operation.

## Decisions

### D1: Layered architecture with an abstract `MemoryBase` (from GreyMagic)

`MemoryBase` defines abstract `ReadBytes`/`WriteBytes`/`Read<T>`/`Write<T>`, relative/absolute addressing, and hosts the `PatchManager`. Two concrete readers:
- `ExternalReader : MemoryBase` — `ReadProcessMemory`/`WriteProcessMemory` over a `SafeMemoryHandle`. Owns allocation, injection, and the remote-thread + main-thread executors.
- `InProcessReader : MemoryBase` — reads the current process via `ReadProcessMemory`/`WriteProcessMemory` on a self-handle; owns the `DetourManager` and `InProcessInvoker`. **(Revised from `unsafe` direct deref during Phase 2: .NET cannot catch `AccessViolationException`, so a bad deref kills the host with no soft-failure path; RPM-on-self fails soft. The in-process speed win moves to the delegate-call/detour paths, not the reader. See `specs/memory-access`.)**

**Why**: GreyMagic proved this abstraction lets the same higher-level code (pattern scan, patch, high-level API) run in either mode. External is the primary path for an automation host; in-process becomes valuable once injected — not for faster reads (both readers use RPM/WPM, see the D1 revision) but for the delegate-call and detour paths it unlocks (`InProcessInvoker`, `DetourManager`).

**Alternatives considered**: single external-only class (current BlackMagic) — rejected: forecloses the in-process delegate path, which is the cleanest crash-free execution. MemorySharp's factory-per-concern model (`Assembly`, `Threads`, `Windows` factories) — adopted selectively for the high-level surface, but the read/write core stays on `MemoryBase` for GreyMagic-style polymorphism.

### D2: Three-tier execution, crash-safe by default (the headline)

Execution is split by **payload safety**, not by convenience:

| Tier | Type | Use when | Mechanism |
|---|---|---|---|
| Remote thread | `RemoteThreadExecutor` | payload is thread-agnostic (LoadLibrary, pure WinAPI, self-contained code payload) | `CreateRemoteThread` + convention stub, wait, exit code |
| **Main-thread pump** | `MainThreadPump` | **payload touches target state** (default) | queue delegate → drained on target thread via per-frame hook |
| In-process | `InProcessInvoker` | injected in-process | `Marshal.GetDelegateForFunctionPointer`, direct call |

`MainThreadPump` installs a detour on a caller-supplied per-frame function address (an `EndScene` resolver ships as a convenience helper) via `DetourManager`. Each frame the hook drains a thread-safe queue and runs pending work items synchronously in the target's context, returning results/exceptions to the requesting thread through a completion handle. **This is net-new — no frame hook exists in current BlackMagic to port.** It is built on GreyMagic-style detours (D5) as its one underlying primitive; the pump is the first consumer of `DetourManager`.

**Why**: `CreateRemoteThread` does not itself crash the target — calling single-thread-affinity target internals from a foreign thread does. Making the pump the default for target-state calls encodes that rule so callers cannot trip the crash by accident, while power users retain raw `CreateRemoteThread` for the payloads it is safe for.

**Alternatives considered**: (a) always `CreateRemoteThread` (MemorySharp/old-BM) — rejected: the documented crash source. (b) always in-process (GreyMagic) — rejected: requires a managed loader in the target and is not always available; external must work standalone. (c) thread-hijack for every call — rejected: high risk, one-shot, poor for repeated calls; kept only for injection.

### D3: `IAssembler` seam replacing FASM

```
IAssembler { byte[] Assemble(string asm, ulong origin = 0); }
```
Two backends:
- `StubAssembler` (default) — hand-emits the fixed calling-convention trampolines (cdecl/stdcall/thiscall/fastcall: push args, call, cleanup, ret) and the injection stubs, using an `EmitU8`/`EmitU32`/`EmitU64` byte emitter (the pattern already in `SInject.cs`). No parsing, deterministic, x64-capable, zero dependency.
- `IcedAssembler` (optional) — wraps Iced for arbitrary user-supplied mnemonics when a caller genuinely needs runtime text assembly.

**Why**: The comparison established that the only thing FASM did was runtime text→bytecode, needed solely for stubs and (rarely) arbitrary asm. Stubs are a small fixed set best hand-emitted; arbitrary asm is better served by a modern, managed, x64, MIT-licensed assembler (Iced) than by a native C++/CLI FASM DLL. Retiring FASM removes the x86-only mixed-mode constraint.

**Alternatives considered**: keep FASM/Fasm.NET — rejected: native dependency, x86-only, unmaintained. Hand-emit everything including arbitrary asm — rejected: reimplementing an assembler is out of scope; Iced already exists.

### D4: `MarshalCache<T>` for typed IO (from GreyMagic)

A `static class MarshalCache<T>` computes and caches `Marshal.SizeOf`, `TypeCode`, `TypeRequiresMarshal`, and `IsIntPtr` once per type. `Read<T>`/`Write<T>` branch on the cached flags: blittable types round-trip through `Span`/`MemoryMarshal`; marshal-required types use `Marshal.PtrToStructure`.

**Why**: GreyMagic's signature perf win — avoids per-call reflection. Current BlackMagic's `where unmanaged` constraint is faster still for blittable types but cannot express marshalled structs; the cache gives both.

### D5: Reversible hooking and patching with auto-restore (from GreyMagic)

`DetourManager`/`Detour` (inline `E9` jmp over a prologue, `CallOriginal`, `Apply`/`Remove`) and `PatchManager`/`Patch` (named byte patch, `Apply`/`Remove`/`IsApplied`). Both register into a manager that restores all live modifications on `MemoryBase.Dispose`. Detours are in-process only (they require executing the hook delegate in the target); patches work in both modes.

**Safety addition**: before splicing a detour, `StubAssembler`/`IcedAssembler` disassembles the target prologue to confirm the overwrite lands on instruction boundaries, avoiding the mid-instruction-splice crash class. When only `StubAssembler` is present, a minimal length-disassembler covers the common prologue shapes; full validation requires the Iced backend.

**Why**: MemorySharp promised Hook/Patch "coming soon" and never shipped them; GreyMagic shipped both and they are the cleanest available. Auto-restore prevents leaving the target corrupted after a crash of the host.

### D6: High-level surface as opt-in factories (from MemorySharp)

`RemotePointer` (indexer `sharp[addr]`), `RemoteModule`/`RemoteFunction` (`sharp["user32"]["MessageBoxA"].Execute(...)`), `ManagedPeb`/`ManagedTeb`, `WindowFactory` (move/resize/title/flash/activate), `Keyboard`/`Mouse` (PostMessage + SendInput), and async wrappers over the executors. These sit above `MemoryBase` and the execution tiers; none is required for core memory work.

**Why**: This is the ergonomic layer that made MemorySharp pleasant. It is pure P/Invoke over the core — no assembler, no FASM.

### D7: New projects, additive, test-first

`WhiteMagic/` (library, `net8.0-windows`, `AllowUnsafeBlocks`) and `WhiteMagicTest/` (xUnit). Task 1.3 creates `WhiteMagic.slnx` (SDK 10's default solution format). The reference libraries live under `reference/` (git-ignored) and are not part of this build. Pure-logic components (assembler encodings, marshal cache, pattern matcher, stub builders, pump queue) get tests before implementation, mirroring the existing `inject-and-assemble` change's TDD discipline. Live-process behavior (RPM/WPM, injection, detours) is validated in integration tests gated on an available target.

## Risks / Trade-offs

- **Scope is large** → Deliver in vertical slices (see Migration Plan). The minimum crash-safe slice is `MemoryBase` + `ExternalReader` + `DetourManager` + `MainThreadPump`; everything else layers on without reworking it.
- **In-process tier needs a managed loader not in scope** → Ship `InProcessReader`'s API and delegate-call path now; mark the actual CLR-host injection as a follow-up. External + pump deliver crash-safety without it.
- **Detour prologue splicing can crash if misaligned** → Validate instruction boundaries before writing (D5); require Iced backend for full validation; keep `Remove`/auto-restore so a bad detour is recoverable.
- **Iced adds a NuGet dependency** → Isolated behind `IAssembler`; default `StubAssembler` keeps the library dependency-free. Only callers needing arbitrary asm opt in.
- **`MainThreadPump` adds per-frame overhead and a hook on a hot function** → Keep the drained-per-frame work bounded; the hook is a thin queue check when idle. Document that a wedged work item stalls the frame.
- **x64 detours need larger/absolute jumps (14-byte `push/ret` or RIP-relative)** → `StubAssembler` emits the correct form per bitness; covered by encoding tests.
- **Blittable vs marshalled ambiguity** in `MarshalCache` → Explicit flags and tests per type category; document that `[StructLayout]` is required for non-blittable remote structs.

## Migration Plan

WhiteMagic is additive; there is nothing to migrate off. Delivery is phased so each slice is independently useful and testable:

1. **Core** — `MemoryBase`, `ExternalReader`, `SafeMemoryHandle`, `MarshalCache`, typed/string/bytes IO. (Replaces nothing; standalone.)
2. **Crash-safe execution slice** — `StubAssembler`, `DetourManager`, `MainThreadPump`, `RemoteThreadExecutor`. Proves state-sensitive calls without crashes end-to-end. This is the headline deliverable.
3. **Injection & discovery** — DLL injection (CreateThread + hijack, x86/x64), pattern scanning + cache, `PeHeaderParser`, named `AllocatedMemory`.
4. **In-process tier** — `InProcessReader`, `InProcessInvoker`, `CreateFunction<T>`, vtable helpers (API surface; managed loader deferred).
5. **High-level ergonomics** — `RemotePointer`, `RemoteModule`/`RemoteFunction`, PEB/TEB, window, input, async, `PatchManager` polish, helpers.
6. **Optional Iced backend** — `IcedAssembler`, arbitrary-asm inject, full prologue validation.

**Rollback**: WhiteMagic is a separate assembly; removing its project reference reverts consumers with no effect on existing libraries.

## Open Questions

- **Frame-hook target**: ~~default to D3D9 `EndScene`, or accept a caller-supplied per-frame function address?~~ **Resolved**: `MainThreadPump` takes a caller-supplied frame-function address (application-agnostic, library stays offset-free per Non-Goals); an `EndScene` resolver ships as a convenience helper only. Slice 2 depends on this — settled before slice 2 starts.
- **Managed in-process loader**: which host mechanism (custom CLR host vs. a native shim that calls `CorBindToRuntimeEx`/`ICLRRuntimeHost`)? Deferred to a follow-up change but affects the `InProcessReader` seam shape.
- **Iced as default vs optional**: keep hand-stubs default (zero dep) — confirmed — but should the library ship a `WhiteMagic.Iced` companion package rather than an optional reference? Package boundary TBD.
- **Async model**: `Task`-based wrappers (MemorySharp) vs. exposing the pump's completion handles directly. Likely both: pump returns a handle, async wrappers adapt it to `Task<T>`.