Add whitemagic-foundation OpenSpec design; isolate reference libs
Design-only foundation for WhiteMagic, a .NET 8 x64 library unifying the four studied process-manipulation libs. Adds proposal, design (7 decisions), 7 capability specs, and TDD task breakdown; all validate strict. Move Blackmagic, Blackmagic-old, GreyMagic, MemorySharp, fasm into reference/ (gitignored) — studied, not built here; each has its own upstream repo and nested .git. Rewrite plan doc paths to reference/. Corrects two factual defects found in review: - current BlackMagic has no D3D EndScene hook; MainThreadPump is net-new built on DetourManager, not a port - no BlackMagic.slnx exists; task 1.3 creates a fresh solution Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -0,0 +1,126 @@
|
||||
## Context
|
||||
|
||||
This repo contains four C# process-manipulation libraries studied in `docs/memory-library-comparison.md`:
|
||||
|
||||
- **BlackMagic-old** (`reference/Blackmagic-old/`) — .NET FW 4.0, x86, FASM as a public `Asm` property.
|
||||
- **MemorySharp** (`reference/MemorySharp/`) — .NET FW, x86, FASM behind `Execute<T>`; rich high-level API (PEB/TEB, window, input, calling conventions) but unmaintained since 2016.
|
||||
- **GreyMagic** (`reference/GreyMagic/`) — .NET FW, ~2016, x86; dual in/out-of-process `MemoryBase`, `MarshalCache`, `DetourManager`, `PatchManager`, `CreateFunction<T>`, `PeHeaderParser`.
|
||||
- **BlackMagic** (`reference/Blackmagic/`, current) — .NET 8, x86 + x64, FASM removed, pattern scanning + cache, DLL injection (CreateThread + hijack), hand-assembled x86 stubs (`SInject.cs` `EmitU8`/`EmitU32` byte emitter), `CreateRemoteThread`-based `Execute` (`BMThread.cs`). No frame hook exists.
|
||||
|
||||
The consuming use case is a WoW 3.3.5a bot. The dominant failure mode observed (`FASM-MIGRATION.md`) is game crashes when protected/game-state functions are invoked on a thread created by `CreateRemoteThread`, because WoW's main thread has exclusive affinity for the Lua VM, D3D9 device, and object manager.
|
||||
|
||||
WhiteMagic is a **new, additive** .NET 8 library that unifies the four. It reuses their ideas, not their assemblies. No project already depends on WhiteMagic, so there is no backward-compatibility constraint.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Single modern (.NET 8, nullable, `Span<byte>`, `SafeHandle`) library that is bitness-agnostic (x86 + x64).
|
||||
- A **three-tier execution model** whose default path for game-state calls is crash-safe (runs on the target's own thread), while `CreateRemoteThread` remains available for thread-agnostic payloads.
|
||||
- Dual memory access: out-of-process (RPM/WPM) and in-process (direct deref) behind one abstract `MemoryBase`, with `MarshalCache<T>` for allocation-free typed IO.
|
||||
- Reversible function hooking (`DetourManager`) and byte patching (`PatchManager`) with auto-restore on dispose.
|
||||
- Replace FASM with an `IAssembler` seam: hand-emitted convention stubs by default, optional Iced backend for arbitrary assembly. Zero native dependency in the default configuration.
|
||||
- Port DLL injection (CreateThread + thread-hijack, x86/x64) and pattern scanning + cache from current BlackMagic.
|
||||
- Provide MemorySharp-grade ergonomics: `RemotePointer` indexer, `Module["fn"].Execute(...)`, PEB/TEB, window mutation, input simulation, async wrappers, helpers.
|
||||
- Test-first for all pure logic (assembler encodings, marshal cache, pattern matching, stub building, pump queue semantics).
|
||||
|
||||
**Non-Goals:**
|
||||
- Modifying or replacing BlackMagic/MemorySharp/GreyMagic — WhiteMagic is additive.
|
||||
- Shipping a full x86/x64 assembler ourselves — arbitrary assembly is delegated to Iced; only the fixed convention-stub shapes are hand-emitted.
|
||||
- Managed-DLL injection bootstrapper (the CLR host that loads `InProcessReader` into the target). WhiteMagic exposes the in-process API surface; wiring an actual managed loader is a follow-up change.
|
||||
- WoW-specific offsets, Lua unlock, or bot logic — those live in the consumer, not the library.
|
||||
- Anti-cheat evasion.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Layered architecture with an abstract `MemoryBase` (from GreyMagic)
|
||||
|
||||
`MemoryBase` defines abstract `ReadBytes`/`WriteBytes`/`Read<T>`/`Write<T>`, relative/absolute addressing, and hosts the `PatchManager`. Two concrete readers:
|
||||
- `ExternalReader : MemoryBase` — `ReadProcessMemory`/`WriteProcessMemory` over a `SafeMemoryHandle`. Owns allocation, injection, and the remote-thread + main-thread executors.
|
||||
- `InProcessReader : MemoryBase` — `unsafe` direct pointer deref; owns the `DetourManager` and `InProcessInvoker`.
|
||||
|
||||
**Why**: GreyMagic proved this abstraction lets the same higher-level code (pattern scan, patch, high-level API) run in either mode. External is the primary path for a bot host; in-process is the fast/crash-free path once injected.
|
||||
|
||||
**Alternatives considered**: single external-only class (current BlackMagic) — rejected: forecloses the in-process delegate path, which is the cleanest crash-free execution. MemorySharp's factory-per-concern model (`Assembly`, `Threads`, `Windows` factories) — adopted selectively for the high-level surface, but the read/write core stays on `MemoryBase` for GreyMagic-style polymorphism.
|
||||
|
||||
### D2: Three-tier execution, crash-safe by default (the headline)
|
||||
|
||||
Execution is split by **payload safety**, not by convenience:
|
||||
|
||||
| Tier | Type | Use when | Mechanism |
|
||||
|---|---|---|---|
|
||||
| Remote thread | `RemoteThreadExecutor` | payload is thread-agnostic (LoadLibrary, pure WinAPI, self-contained shellcode) | `CreateRemoteThread` + convention stub, wait, exit code |
|
||||
| **Main-thread pump** | `MainThreadPump` | **payload touches game state** (default) | queue delegate → drained on target thread via per-frame hook |
|
||||
| In-process | `InProcessInvoker` | injected in-process | `Marshal.GetDelegateForFunctionPointer`, direct call |
|
||||
|
||||
`MainThreadPump` installs a detour on a caller-supplied per-frame function address (an `EndScene` resolver ships as a convenience helper) via `DetourManager`. Each frame the hook drains a thread-safe queue and runs pending work items synchronously in the game's context, returning results/exceptions to the requesting thread through a completion handle. **This is net-new — no frame hook exists in current BlackMagic to port.** It is built on GreyMagic-style detours (D5) as its one underlying primitive; the pump is the first consumer of `DetourManager`.
|
||||
|
||||
**Why**: `CreateRemoteThread` does not itself crash the game — calling single-thread-affinity game internals from a foreign thread does. Making the pump the default for game calls encodes that rule so callers cannot trip the crash by accident, while power users retain raw `CreateRemoteThread` for the payloads it is safe for.
|
||||
|
||||
**Alternatives considered**: (a) always `CreateRemoteThread` (MemorySharp/old-BM) — rejected: the documented crash source. (b) always in-process (GreyMagic) — rejected: requires a managed loader in the target and is not always available; external must work standalone. (c) thread-hijack for every call — rejected: high risk, one-shot, poor for repeated calls; kept only for injection.
|
||||
|
||||
### D3: `IAssembler` seam replacing FASM
|
||||
|
||||
```
|
||||
IAssembler { byte[] Assemble(string asm, ulong origin = 0); }
|
||||
```
|
||||
Two backends:
|
||||
- `StubAssembler` (default) — hand-emits the fixed calling-convention trampolines (cdecl/stdcall/thiscall/fastcall: push args, call, cleanup, ret) and the injection stubs, using an `EmitU8`/`EmitU32`/`EmitU64` byte emitter (the pattern already in `SInject.cs`). No parsing, deterministic, x64-capable, zero dependency.
|
||||
- `IcedAssembler` (optional) — wraps Iced for arbitrary user-supplied mnemonics when a caller genuinely needs runtime text assembly.
|
||||
|
||||
**Why**: The comparison established that the only thing FASM did was runtime text→bytecode, needed solely for stubs and (rarely) arbitrary asm. Stubs are a small fixed set best hand-emitted; arbitrary asm is better served by a modern, managed, x64, MIT-licensed assembler (Iced) than by a native C++/CLI FASM DLL. Retiring FASM removes the x86-only mixed-mode constraint.
|
||||
|
||||
**Alternatives considered**: keep FASM/Fasm.NET — rejected: native dependency, x86-only, unmaintained. Hand-emit everything including arbitrary asm — rejected: reimplementing an assembler is out of scope; Iced already exists.
|
||||
|
||||
### D4: `MarshalCache<T>` for typed IO (from GreyMagic)
|
||||
|
||||
A `static class MarshalCache<T>` computes and caches `Marshal.SizeOf`, `TypeCode`, `TypeRequiresMarshal`, and `IsIntPtr` once per type. `Read<T>`/`Write<T>` branch on the cached flags: blittable types round-trip through `Span`/`MemoryMarshal`; marshal-required types use `Marshal.PtrToStructure`.
|
||||
|
||||
**Why**: GreyMagic's signature perf win — avoids per-call reflection. Current BlackMagic's `where unmanaged` constraint is faster still for blittable types but cannot express marshalled structs; the cache gives both.
|
||||
|
||||
### D5: Reversible hooking and patching with auto-restore (from GreyMagic)
|
||||
|
||||
`DetourManager`/`Detour` (inline `E9` jmp over a prologue, `CallOriginal`, `Apply`/`Remove`) and `PatchManager`/`Patch` (named byte patch, `Apply`/`Remove`/`IsApplied`). Both register into a manager that restores all live modifications on `MemoryBase.Dispose`. Detours are in-process only (they require executing the hook delegate in the target); patches work in both modes.
|
||||
|
||||
**Safety addition**: before splicing a detour, `StubAssembler`/`IcedAssembler` disassembles the target prologue to confirm the overwrite lands on instruction boundaries, avoiding the mid-instruction-splice crash class. When only `StubAssembler` is present, a minimal length-disassembler covers the common prologue shapes; full validation requires the Iced backend.
|
||||
|
||||
**Why**: MemorySharp promised Hook/Patch "coming soon" and never shipped them; GreyMagic shipped both and they are the cleanest available. Auto-restore prevents leaving the target corrupted after a crash of the host.
|
||||
|
||||
### D6: High-level surface as opt-in factories (from MemorySharp)
|
||||
|
||||
`RemotePointer` (indexer `sharp[addr]`), `RemoteModule`/`RemoteFunction` (`sharp["user32"]["MessageBoxA"].Execute(...)`), `ManagedPeb`/`ManagedTeb`, `WindowFactory` (move/resize/title/flash/activate), `Keyboard`/`Mouse` (PostMessage + SendInput), and async wrappers over the executors. These sit above `MemoryBase` and the execution tiers; none is required for core memory work.
|
||||
|
||||
**Why**: This is the ergonomic layer that made MemorySharp pleasant. It is pure P/Invoke over the core — no assembler, no FASM.
|
||||
|
||||
### D7: New projects, additive, test-first
|
||||
|
||||
`WhiteMagic/` (library, `net8.0-windows`, `AllowUnsafeBlocks`) and `WhiteMagicTest/` (xUnit). No solution file exists in the repo root today; task 1.3 creates a fresh `WhiteMagic.sln` (or builds the csproj directly). BlackMagic lives in a nested subdir with its own git and is not part of this build. Pure-logic components (assembler encodings, marshal cache, pattern matcher, stub builders, pump queue) get tests before implementation, mirroring the existing `inject-and-assemble` change's TDD discipline. Live-process behavior (RPM/WPM, injection, detours) is validated in integration tests gated on an available target.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **Scope is large** → Deliver in vertical slices (see Migration Plan). The minimum crash-safe slice is `MemoryBase` + `ExternalReader` + `DetourManager` + `MainThreadPump`; everything else layers on without reworking it.
|
||||
- **In-process tier needs a managed loader not in scope** → Ship `InProcessReader`'s API and delegate-call path now; mark the actual CLR-host injection as a follow-up. External + pump deliver crash-safety without it.
|
||||
- **Detour prologue splicing can crash if misaligned** → Validate instruction boundaries before writing (D5); require Iced backend for full validation; keep `Remove`/auto-restore so a bad detour is recoverable.
|
||||
- **Iced adds a NuGet dependency** → Isolated behind `IAssembler`; default `StubAssembler` keeps the library dependency-free. Only callers needing arbitrary asm opt in.
|
||||
- **`MainThreadPump` adds per-frame overhead and a hook on a hot function** → Keep the drained-per-frame work bounded; the hook is a thin queue check when idle. Document that a wedged work item stalls the frame.
|
||||
- **x64 detours need larger/absolute jumps (14-byte `push/ret` or RIP-relative)** → `StubAssembler` emits the correct form per bitness; covered by encoding tests.
|
||||
- **Blittable vs marshalled ambiguity** in `MarshalCache` → Explicit flags and tests per type category; document that `[StructLayout]` is required for non-blittable remote structs.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
WhiteMagic is additive; there is nothing to migrate off. Delivery is phased so each slice is independently useful and testable:
|
||||
|
||||
1. **Core** — `MemoryBase`, `ExternalReader`, `SafeMemoryHandle`, `MarshalCache`, typed/string/bytes IO. (Replaces nothing; standalone.)
|
||||
2. **Crash-safe execution slice** — `StubAssembler`, `DetourManager`, `MainThreadPump`, `RemoteThreadExecutor`. Proves game-state calls without crashes end-to-end. This is the headline deliverable.
|
||||
3. **Injection & discovery** — DLL injection (CreateThread + hijack, x86/x64), pattern scanning + cache, `PeHeaderParser`, named `AllocatedMemory`.
|
||||
4. **In-process tier** — `InProcessReader`, `InProcessInvoker`, `CreateFunction<T>`, vtable helpers (API surface; managed loader deferred).
|
||||
5. **High-level ergonomics** — `RemotePointer`, `RemoteModule`/`RemoteFunction`, PEB/TEB, window, input, async, `PatchManager` polish, helpers.
|
||||
6. **Optional Iced backend** — `IcedAssembler`, arbitrary-asm inject, full prologue validation.
|
||||
|
||||
**Rollback**: WhiteMagic is a separate assembly; removing its project reference reverts consumers with no effect on existing libraries.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Frame-hook target**: ~~default to D3D9 `EndScene`, or accept a caller-supplied per-frame function address?~~ **Resolved**: `MainThreadPump` takes a caller-supplied frame-function address (WoW-agnostic, library stays offset-free per Non-Goals); an `EndScene` resolver ships as a convenience helper only. Slice 2 depends on this — settled before slice 2 starts.
|
||||
- **Managed in-process loader**: which host mechanism (custom CLR host vs. a native shim that calls `CorBindToRuntimeEx`/`ICLRRuntimeHost`)? Deferred to a follow-up change but affects the `InProcessReader` seam shape.
|
||||
- **Iced as default vs optional**: keep hand-stubs default (zero dep) — confirmed — but should the library ship a `WhiteMagic.Iced` companion package rather than an optional reference? Package boundary TBD.
|
||||
- **Async model**: `Task`-based wrappers (MemorySharp) vs. exposing the pump's completion handles directly. Likely both: pump returns a handle, async wrappers adapt it to `Task<T>`.
|
||||
Reference in New Issue
Block a user