whitemagic/openspec/changes/whitemagic-foundation/design.md

Context

This repo contains four C# process-manipulation libraries studied in docs/memory-library-comparison.md:

• BlackMagic-old (reference/Blackmagic-old/) — .NET FW 4.0, x86, FASM as a public Asm property.
• MemorySharp (reference/MemorySharp/) — .NET FW, x86, FASM behind Execute<T>; rich high-level API (PEB/TEB, window, input, calling conventions) but unmaintained since 2016.
• GreyMagic (reference/GreyMagic/) — .NET FW, ~2016, x86; dual in/out-of-process MemoryBase, MarshalCache, DetourManager, PatchManager, CreateFunction<T>, PeHeaderParser.
• BlackMagic (reference/Blackmagic/, current) — .NET 8, x86 + x64, FASM removed, pattern scanning + cache, DLL injection (CreateThread + hijack), hand-assembled x86 stubs (SInject.cs EmitU8/EmitU32 byte emitter), CreateRemoteThread-based Execute (BMThread.cs). No frame hook exists.

The consuming use case is an automation client targeting a legacy x86 desktop application. The dominant failure mode observed (FASM-MIGRATION.md) is target crashes when state-sensitive functions are invoked on a thread created by CreateRemoteThread, because the target's main thread has exclusive affinity for its scripting VM, render device, and object model.

WhiteMagic is a new, additive .NET 8 library that unifies the four. It reuses their ideas, not their assemblies. No project already depends on WhiteMagic, so there is no backward-compatibility constraint.

Goals / Non-Goals

Goals:

• Single modern (.NET 8, nullable, Span<byte>, SafeHandle) library that is bitness-agnostic (x86 + x64).
• A three-tier execution model whose default path for state-sensitive calls is crash-safe (runs on the target's own thread), while CreateRemoteThread remains available for thread-agnostic payloads.
• Dual memory access: out-of-process (RPM/WPM) and in-process (RPM-on-self-handle, see D1 revision) behind one abstract MemoryBase, with MarshalCache<T> for allocation-free typed IO.
• Reversible function hooking (DetourManager) and byte patching (PatchManager) with auto-restore on dispose.
• Replace FASM with an IAssembler seam: hand-emitted convention stubs by default, optional Iced backend for arbitrary assembly. Zero native dependency in the default configuration.
• Port DLL injection (CreateThread + thread-hijack, x86/x64) and pattern scanning + cache from current BlackMagic.
• Provide MemorySharp-grade ergonomics: RemotePointer indexer, Module["fn"].Execute(...), PEB/TEB, window mutation, input simulation, async wrappers, helpers.
• Test-first for all pure logic (assembler encodings, marshal cache, pattern matching, stub building, pump queue semantics).

Non-Goals:

• Modifying or replacing BlackMagic/MemorySharp/GreyMagic — WhiteMagic is additive.
• Shipping a full x86/x64 assembler ourselves — arbitrary assembly is delegated to Iced; only the fixed convention-stub shapes are hand-emitted.
• Managed-DLL injection bootstrapper (the CLR host that loads InProcessReader into the target). WhiteMagic exposes the in-process API surface; wiring an actual managed loader is a follow-up change.
• Application-specific offsets, scripting-engine hooks, or automation logic — those live in the consumer, not the library.
• Interference with other software's operation.

Decisions

D1: Layered architecture with an abstract MemoryBase (from GreyMagic)

MemoryBase defines abstract ReadBytes/WriteBytes/Read<T>/Write<T>, relative/absolute addressing, and hosts the PatchManager. Two concrete readers:

• ExternalReader : MemoryBase — ReadProcessMemory/WriteProcessMemory over a SafeMemoryHandle. Owns allocation, injection, and the remote-thread + main-thread executors.
• InProcessReader : MemoryBase — reads the current process via ReadProcessMemory/WriteProcessMemory on a self-handle; owns the DetourManager and InProcessInvoker. (Revised from unsafe direct deref during Phase 2: .NET cannot catch AccessViolationException, so a bad deref kills the host with no soft-failure path; RPM-on-self fails soft. The in-process speed win moves to the delegate-call/detour paths, not the reader. See specs/memory-access.)

Why: GreyMagic proved this abstraction lets the same higher-level code (pattern scan, patch, high-level API) run in either mode. External is the primary path for an automation host; in-process becomes valuable once injected — not for faster reads (both readers use RPM/WPM, see the D1 revision) but for the delegate-call and detour paths it unlocks (InProcessInvoker, DetourManager).

Alternatives considered: single external-only class (current BlackMagic) — rejected: forecloses the in-process delegate path, which is the cleanest crash-free execution. MemorySharp's factory-per-concern model (Assembly, Threads, Windows factories) — adopted selectively for the high-level surface, but the read/write core stays on MemoryBase for GreyMagic-style polymorphism.

D2: Three-tier execution, crash-safe by default (the headline)

Execution is split by payload safety, not by convenience:

Tier	Type	Use when	Mechanism
Remote thread	RemoteThreadExecutor	payload is thread-agnostic (LoadLibrary, pure WinAPI, self-contained code payload)	CreateRemoteThread + convention stub, wait, exit code
Main-thread pump	MainThreadPump	payload touches target state (default)	queue delegate → drained on target thread via per-frame hook
In-process	InProcessInvoker	injected in-process	Marshal.GetDelegateForFunctionPointer, direct call

MainThreadPump installs a detour on a caller-supplied per-frame function address (an EndScene resolver ships as a convenience helper) via DetourManager. Each frame the hook drains a thread-safe queue and runs pending work items synchronously in the target's context, returning results/exceptions to the requesting thread through a completion handle. This is net-new — no frame hook exists in current BlackMagic to port. It is built on GreyMagic-style detours (D5) as its one underlying primitive; the pump is the first consumer of DetourManager.

Why: CreateRemoteThread does not itself crash the target — calling single-thread-affinity target internals from a foreign thread does. Making the pump the default for target-state calls encodes that rule so callers cannot trip the crash by accident, while power users retain raw CreateRemoteThread for the payloads it is safe for.

Alternatives considered: (a) always CreateRemoteThread (MemorySharp/old-BM) — rejected: the documented crash source. (b) always in-process (GreyMagic) — rejected: requires a managed loader in the target and is not always available; external must work standalone. (c) thread-hijack for every call — rejected: high risk, one-shot, poor for repeated calls; kept only for injection.

D3: IAssembler seam replacing FASM

IAssembler { byte[] Assemble(string asm, ulong origin = 0); }

Two backends:

• StubAssembler (default) — hand-emits the fixed calling-convention trampolines (cdecl/stdcall/thiscall/fastcall: push args, call, cleanup, ret) and the injection stubs, using an EmitU8/EmitU32/EmitU64 byte emitter (the pattern already in SInject.cs). No parsing, deterministic, x64-capable, zero dependency.
• IcedAssembler (optional) — wraps Iced for arbitrary user-supplied mnemonics when a caller genuinely needs runtime text assembly.

Why: The comparison established that the only thing FASM did was runtime text→bytecode, needed solely for stubs and (rarely) arbitrary asm. Stubs are a small fixed set best hand-emitted; arbitrary asm is better served by a modern, managed, x64, MIT-licensed assembler (Iced) than by a native C++/CLI FASM DLL. Retiring FASM removes the x86-only mixed-mode constraint.

Alternatives considered: keep FASM/Fasm.NET — rejected: native dependency, x86-only, unmaintained. Hand-emit everything including arbitrary asm — rejected: reimplementing an assembler is out of scope; Iced already exists.

D4: MarshalCache<T> for typed IO (from GreyMagic)

A static class MarshalCache<T> computes and caches Marshal.SizeOf, TypeCode, TypeRequiresMarshal, and IsIntPtr once per type. Read<T>/Write<T> branch on the cached flags: blittable types round-trip through Span/MemoryMarshal; marshal-required types use Marshal.PtrToStructure.

Why: GreyMagic's signature perf win — avoids per-call reflection. Current BlackMagic's where unmanaged constraint is faster still for blittable types but cannot express marshalled structs; the cache gives both.

D5: Reversible hooking and patching with auto-restore (from GreyMagic)

DetourManager/Detour (inline E9 jmp over a prologue, CallOriginal, Apply/Remove) and PatchManager/Patch (named byte patch, Apply/Remove/IsApplied). Both register into a manager that restores all live modifications on MemoryBase.Dispose. Detours are in-process only (they require executing the hook delegate in the target); patches work in both modes.

Safety addition: before splicing a detour, StubAssembler/IcedAssembler disassembles the target prologue to confirm the overwrite lands on instruction boundaries, avoiding the mid-instruction-splice crash class. When only StubAssembler is present, a minimal length-disassembler covers the common prologue shapes; full validation requires the Iced backend.

Why: MemorySharp promised Hook/Patch "coming soon" and never shipped them; GreyMagic shipped both and they are the cleanest available. Auto-restore prevents leaving the target corrupted after a crash of the host.

D6: High-level surface as opt-in factories (from MemorySharp)

RemotePointer (indexer sharp[addr]), RemoteModule/RemoteFunction (sharp["user32"]["MessageBoxA"].Execute(...)), ManagedPeb/ManagedTeb, WindowFactory (move/resize/title/flash/activate), Keyboard/Mouse (PostMessage + SendInput), and async wrappers over the executors. These sit above MemoryBase and the execution tiers; none is required for core memory work.

Why: This is the ergonomic layer that made MemorySharp pleasant. It is pure P/Invoke over the core — no assembler, no FASM.

D7: New projects, additive, test-first

WhiteMagic/ (library, net8.0-windows, AllowUnsafeBlocks) and WhiteMagicTest/ (xUnit). Task 1.3 creates WhiteMagic.slnx (SDK 10's default solution format). The reference libraries live under reference/ (git-ignored) and are not part of this build. Pure-logic components (assembler encodings, marshal cache, pattern matcher, stub builders, pump queue) get tests before implementation, mirroring the existing inject-and-assemble change's TDD discipline. Live-process behavior (RPM/WPM, injection, detours) is validated in integration tests gated on an available target.

Risks / Trade-offs

• Scope is large → Deliver in vertical slices (see Migration Plan). The minimum crash-safe slice is MemoryBase + ExternalReader + DetourManager + MainThreadPump; everything else layers on without reworking it.
• In-process tier needs a managed loader not in scope → Ship InProcessReader's API and delegate-call path now; mark the actual CLR-host injection as a follow-up. External + pump deliver crash-safety without it.
• Detour prologue splicing can crash if misaligned → Validate instruction boundaries before writing (D5); require Iced backend for full validation; keep Remove/auto-restore so a bad detour is recoverable.
• Iced adds a NuGet dependency → Isolated behind IAssembler; default StubAssembler keeps the library dependency-free. Only callers needing arbitrary asm opt in.
• MainThreadPump adds per-frame overhead and a hook on a hot function → Keep the drained-per-frame work bounded; the hook is a thin queue check when idle. Document that a wedged work item stalls the frame.
• x64 detours need larger/absolute jumps (14-byte push/ret or RIP-relative) → StubAssembler emits the correct form per bitness; covered by encoding tests.
• Blittable vs marshalled ambiguity in MarshalCache → Explicit flags and tests per type category; document that [StructLayout] is required for non-blittable remote structs.

Migration Plan

WhiteMagic is additive; there is nothing to migrate off. Delivery is phased so each slice is independently useful and testable:

1. Core — MemoryBase, ExternalReader, SafeMemoryHandle, MarshalCache, typed/string/bytes IO. (Replaces nothing; standalone.)
2. Crash-safe execution slice — StubAssembler, DetourManager, MainThreadPump, RemoteThreadExecutor. Proves state-sensitive calls without crashes end-to-end. This is the headline deliverable.
3. Injection & discovery — DLL injection (CreateThread + hijack, x86/x64), pattern scanning + cache, PeHeaderParser, named AllocatedMemory.
4. In-process tier — InProcessReader, InProcessInvoker, CreateFunction<T>, vtable helpers (API surface; managed loader deferred).
5. High-level ergonomics — RemotePointer, RemoteModule/RemoteFunction, PEB/TEB, window, input, async, PatchManager polish, helpers.
6. Optional Iced backend — IcedAssembler, arbitrary-asm inject, full prologue validation.

Rollback: WhiteMagic is a separate assembly; removing its project reference reverts consumers with no effect on existing libraries.

Open Questions

• Frame-hook target: default to D3D9 EndScene, or accept a caller-supplied per-frame function address? Resolved: MainThreadPump takes a caller-supplied frame-function address (application-agnostic, library stays offset-free per Non-Goals); an EndScene resolver ships as a convenience helper only. Slice 2 depends on this — settled before slice 2 starts.
• Managed in-process loader: which host mechanism (custom CLR host vs. a native shim that calls CorBindToRuntimeEx/ICLRRuntimeHost)? Deferred to a follow-up change but affects the InProcessReader seam shape.
• Iced as default vs optional: keep hand-stubs default (zero dep) — confirmed — but should the library ship a WhiteMagic.Iced companion package rather than an optional reference? Package boundary TBD.
• Async model: Task-based wrappers (MemorySharp) vs. exposing the pump's completion handles directly. Likely both: pump returns a handle, async wrappers adapt it to Task<T>.