whitemagic/docs/memory-library-comparison.md

# Process-Introspection Library Comparison — BlackMagic-old, MemorySharp, GreyMagic, BlackMagic

**Date**: 2026-07-21
**Purpose**: Compare four C# process-introspection libraries present in this repo and derive the design of a modern successor ("WhiteMagic"). The OpenSpec change `whitemagic-foundation` formalizes the design; this document is the supporting study.

## The four subjects

| Library | Location | Era / Platform | Role in this study |
|---|---|---|---|
| **BlackMagic-old** | `reference/Blackmagic-old/` | .NET FW 4.0, x86 | The FASM "before" — initial commit, FASM as public API |
| **MemorySharp** | `reference/MemorySharp/` (≡ `lib/MemorySharp/`, byte-identical) | .NET FW, x86 | Contemporaneous peer that kept FASM behind a polished API |
| **GreyMagic** | `reference/GreyMagic/` | .NET FW, ~2016, x86 | Introduces the in-process execution model + detour/patch/marshal-cache |
| **BlackMagic** (current) | `reference/Blackmagic/` | **.NET 8, x86 + x64** | The FASM "after" — modernized, FASM removed |

> `reference/MemorySharp/` and `lib/MemorySharp/` are identical copies (`diff -rq` clean).

## Where FASM lives (the through-line)

FASM (the Flat Assembler, via the `ManagedFasm` C++/CLI wrapper in `reference/fasm/`) exists in these libraries for exactly one job: **turning assembly text into machine code at runtime**, so a small call-stub / injection-stub can be synthesized when the target address, arguments, and calling convention are only known at call time.

- **BlackMagic-old** — FASM is a *public, load-bearing* dependency: `public ManagedFasm Asm { get; set; }` sits directly on the facade (`BMMain.cs:80`), and `SInject.cs` builds its DLL-redirect injection stub as mnemonic text at runtime.
- **MemorySharp** — same dependency, *wrapped*: the assembler is internal (`Fasm32Assembler`, created unconditionally by `AssemblyFactory`), driven by calling-convention formatters behind `Execute<T>`.
- **GreyMagic** — FASM only on the *external* path (`ExternalProcessReader.Asm`); the in-process path needs no assembler because it calls functions as delegates directly.
- **BlackMagic (current)** — FASM **removed entirely**. Injection stubs became compile-time `byte[]` (`BuildStub32/64`); the public `Asm` property was deleted; runtime target execution moved to the D3D EndScene hook. See `FASM-MIGRATION.md`.

**Conclusion**: the assembly subsystem is not inherent to process introspection — it is a consequence of choosing `CreateRemoteThread` + "support arbitrary calling conventions" as the execution contract. Change the execution primitive (as current BM did) and the need for a runtime assembler evaporates. A managed assembler ([Iced](https://github.com/icedland/iced)) or hand-emitted stubs cover the residual need with no native dependency and full x64 support.

## Feature matrix

| Axis | BM-old | MemorySharp | GreyMagic | BM (current) |
|---|---|---|---|---|
| Platform | FW 4.0, x86 | FW, x86 | FW, x86 | **.NET 8, x86 + x64** |
| Handles | raw `IntPtr` | `SafeMemoryHandle` | `SafeMemoryHandle` | `SafeMemoryHandle`, nullable |
| Addressing | `uint` | `IntPtr` | `IntPtr` | `IntPtr` (64-bit-safe) |
| Process model | external | external | **external + in-process** | external (+ frame-hook) |
| Typed read/write | `ReadInt` etc. | `Read<T>` + marshal | `Read<T>` + **MarshalCache** | `Read<T> where unmanaged` |
| Pattern scanning | ✅ | ❌ ("coming soon") | ❌ (has PE parser) | ✅ + cache |
| FASM / assembler | **public `Asm`** | internal, behind `Execute<T>` | `Asm` (external only) | **none** |
| Remote fn call | raw `Asm` stubs | **`Execute<T>(conv, args…)`** + async | **in-proc delegates** | D3D frame-hook stub |
| Function hooking | — | — | **Detour mgr** (reversible, `CallOriginal`) | Frame-hook only |
| Byte patching | — | ❌ ("coming soon") | **Patch mgr** (named, reversible) | ad hoc |
| DLL injection | CreateThread + hijack | LoadLibrary via CreateThread | (via `Asm`) | CreateThread + hijack, x86/x64 |
| Named allocation | — | `RemoteAllocation` | **`AllocatedMemory`** (by name) | `AllocateMemory` |
| PE parsing | — | — | **`PeHeaderParser`** | — |
| PEB / TEB | — | ✅ `ManagedPeb`/`ManagedTeb` | — | — |
| Window mutation | — | ✅ (move/resize/title/flash) | — | — |
| Input simulation | — | ✅ (keyboard/mouse, no focus) | — | — |
| High-level ergonomics | simple facade | `sharp[addr]`, `module["fn"]`, Enum reads | `CreateFunction<T>`, vtable helpers | facade |
| Helpers | — | ApplicationFinder, HandleManipulator, Randomizer, Serialization, Singleton | MarshalCache, Utilities | — |

## What each does best (the "take from each" summary)

- **BlackMagic-old** → the clean minimal `Open / Read / Write / FindPattern` facade; it is also the historical proof that FASM was once load-bearing and can be retired.
- **MemorySharp** → high-level ergonomics: calling-convention `Execute<T>` + parameter marshalling + async, `RemotePointer` indexer, PEB/TEB, window + keyboard/mouse simulation, helper utilities.
- **GreyMagic** → the engine: dual in/out-of-process `MemoryBase`, `MarshalCache<T>` fast typed IO, reversible `DetourManager` + `PatchManager`, `CreateFunction<T>`/vtable helpers, named `AllocatedMemory`, `PeHeaderParser`.
- **BlackMagic (current)** → the modern platform: .NET 8, `SafeMemoryHandle`, nullable, `Span<byte>`, **x64**, pattern scanning + cache, rich DLL injection (CreateThread + thread-hijack, x86/x64 stubs), hand-assembled stubs (no FASM), per-frame hook (crash-safe execution), test coverage.

## The crash-safety principle (why `CreateRemoteThread` needs care)

`CreateRemoteThread` does not crash the target — **calling target internals from the wrong thread does**. The target's main thread has exclusive affinity for its scripting VM, the render device, and the object model. A thread you spawn runs concurrently with it; the moment a payload touches that state (scripting-engine entry points, object traversal) it races the main thread → memory corruption → crash. This matches the "3 crashes in one session" recorded in `FASM-MIGRATION.md`.

The rule the successor must encode — **split execution by payload safety**:

1. **`CreateRemoteThread` is safe** for *self-contained, thread-agnostic* payloads: `LoadLibrary` (DLL injection), pure WinAPI, code touching only memory you own.
2. **State-sensitive calls must run on the target's own thread**, reached by hooking a per-frame function (D3D `EndScene`, or any frame function via a detour) and draining a work queue there each frame.
3. **In-process** (once a managed DLL is injected), call target functions directly as delegates — no thread crossing at all.

## WhiteMagic — synthesis

A modern successor unifying the four. Full design in `openspec/changes/whitemagic-foundation/`.

```
WhiteMagic (facade — BM-old ergonomics)
├─ Core:        SafeHandle, native P/Invoke, x64             [BM current]
├─ MemoryBase (abstract Read/Write + MarshalCache)           [GreyMagic]
│   ├─ ExternalReader (RPM/WPM)
│   └─ InProcessReader (RPM/WPM on self-handle, injected)
├─ Discovery:   PatternScanner(+cache), PeHeaderParser       [BM current + GreyMagic]
├─ Allocation:  AllocatedMemory (named chunks)               [GreyMagic]
├─ Assembler:   IAssembler → { HandStubs | Iced }            [BM current; Iced replaces FASM]
├─ Execution (three tiers):
│   ├─ RemoteThreadExecutor  (CreateRemoteThread — safe payloads)   [MemorySharp Execute<T>, no FASM]
│   ├─ MainThreadPump        (frame-hook work queue)               [BM frame-hook + GreyMagic detour]  ← crash-safe
│   └─ InProcessInvoker      (CreateFunction<T> delegates)          [GreyMagic]
├─ Hooking:     DetourManager + PatchManager                 [GreyMagic]
├─ Injection:   CreateThread + ThreadHijack (x86/x64)        [BM current]
├─ HighLevel:   RemotePointer, Module["fn"], PEB/TEB,
│               input sim, window, async, helpers            [MemorySharp]
└─ Safety:      disassemble-before-splice (Iced), auto-restore
                all patches/detours on Dispose               [new]
```

**Net result**: BM's modern, FASM-free, x64 core + GreyMagic's dual-mode / detour / patch / marshal-cache engine + MemorySharp's high-level ergonomics — with a three-tier execution model whose *default* for state-sensitive calls is the crash-safe main-thread pump, while `CreateRemoteThread` stays available for the payloads it is genuinely safe for.

### Deviations discovered during implementation

The design held, but building it surfaced corrections worth recording (each is detailed against its task in `openspec/changes/whitemagic-foundation/tasks.md`):

- **`InProcessReader` reads via RPM/WPM on a self-handle, not `unsafe` direct deref** — .NET cannot catch `AccessViolationException`, so a bad direct deref kills the host with no soft-failure path. The in-process speed win moves to the delegate-call and detour paths, not the reader (design decision D1, revised mid-Phase 2).
- **`MarshalCache<T>` splits `Size` (managed, blittable) from `MarshalSize` (`Marshal.SizeOf`, marshal path)** — a single size mis-sized structs whose unmanaged width differs (a `bool` field is managed-1 / unmanaged-4; inline `ByValTStr`/`ByValArray` under-sized the marshal buffer and corrupted the heap on write). `MemoryBase` picks per `TypeRequiresMarshal` at every IO site.
- **x64 call stub is fully MS-x64-ABI compliant** — 32-byte shadow space, 16-byte alignment at the inner `call`, full `imm64` register loads (no >4 GiB pointer truncation), stack args above the shadow window. Proven at runtime by a live SSE callee whose aligned `movaps` faults on any misalignment (task 3.8), not just by byte-level encoding tests.
- **`RemoteModule`/`RemoteFunction` follow PE export forwarders** — `kernel32!HeapAlloc` → `NTDLL.RtlAllocateHeap` and similar resolve into the real target module; ordinal and API-set forwarders throw `NotSupportedException` rather than returning a wrong address (task 7.2).
- **Detour prologue safety is tiered** — the default `StubAssembler` length-decoder covers only the common x86/x64 prologue shapes and refuses any opcode outside that set (zero dependency); the optional `IcedAssembler.GetPrologueLength` decodes arbitrary prologues and is plugged in via `DetourManager.PrologueLengthResolver` when full validation is wanted (tasks 4.6, 8.3).
- **Iced has no text parser** — the design assumed arbitrary text assembly could be delegated to Iced, but Iced ships only a *fluent* code assembler and a decoder. `IcedAssembler.Assemble` bridges Intel-syntax text onto the fluent API by reflection (registers, immediates, labels; memory operands unsupported), rather than depending on a parser that does not exist (task 8.2).
- **Injection bitness corrections** — the thread-hijack injector enforces matching host/target bitness, so the 32-bit path always runs from a 32-bit caller and uses native `GetThreadContext`/`SetThreadContext`; the WOW64 context APIs (for 64-bit callers inspecting WOW64 targets) never apply here and were removed. `ExternalReader` validates `QueryInformation`/`QueryLimitedInformation` access and surfaces `IsWow64Process` failures instead of silently assuming host bitness.
- **Bounds and protection hardening** — `AllocatedMemory` range-checks typed IO against region size; `Patch` mirrors the detour's `VirtualProtectEx` dance; `MainThreadPump` guards the completion race on an already-completed `TaskCompletionSource`.