Fix x64 stub ABI and marshal-path sizing; dedupe memory readers

x64 call stub was ABI-broken: fixed 0x20 frame left rsp misaligned at the
inner call (callee entry rsp ≡ 0, ABI requires ≡ 8) and, for 5+ args, wrote
stack args over the return address. Compute frame K ≡ 8 (mod 16), K ≥
0x20 + 8*stackArgs, so the callee sees a 16-aligned stack and stack args land
above the shadow window. Load register args as full 64-bit imm64 (was imm32,
which truncated pointers > 4 GiB). BuildCallStub now takes nuint[]; x86 range-
checks each arg against uint.MaxValue instead of silently truncating.

MarshalCache conflated managed and unmanaged width in one Size field: the
blittable path needs Unsafe.SizeOf<T> (bool = 1) while the marshal path needs
Marshal.SizeOf<T> (inline ByValTStr/ByValArray expand past the managed
pointer). Add MarshalSize; MemoryBase picks per TypeRequiresMarshal at all four
IO sites. Prevents PtrToStructure/StructureToPtr from over-reading/overwriting
the pinned scratch buffer (heap corruption on write).

Extract shared RPM/WPM into RpmHelper: honor partial reads (dead Array.Resize
removed), consistent write-return semantics; InProcessReader now guards
MainModule like ExternalReader.

Tests: x64 frame-alignment property + inline-marshal round-trip added (both
fail against the pre-fix code); existing x64 byte-expectation tests updated to
the new frame. Build clean, 100/100 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
kbe
2026-07-21 22:30:10 +02:00
co-authored by Claude Opus 4.8
parent 184dec86ca
commit 12b9b6c03e
7 changed files with 343 additions and 101 deletions
+48 -18
View File
@@ -5,25 +5,44 @@ using System.Runtime.InteropServices;
namespace WhiteMagic;
/// <summary>
/// Computes and caches the byte size and marshalling decision for type
/// <typeparamref name="T"/> exactly once. <see cref="MemoryBase.Read{T}"/> and
/// Caches the widths marshalling decisions for type <typeparamref name="T"/>
/// once, at static-constructor time. <see cref="MemoryBase.Read{T}"/> and
/// <see cref="MemoryBase.Write{T}"/> branch on <see cref="TypeRequiresMarshal"/>
/// to decide between the blittable <c>Span</c> /
/// <see cref="System.Runtime.InteropServices.MemoryMarshal"/> path and the
/// <see cref="Marshal.PtrToStructure"/> path.
/// and pick the appropriate width from this cache.
/// </summary>
/// <typeparam name="T">The type to cache metadata for.</typeparam>
public static class MarshalCache<T>
{
/// <summary>
/// The byte size of <typeparamref name="T"/>. For the blittable path this is
/// the managed layout size <see cref="Unsafe.SizeOf{T}"/> — the width that
/// <see cref="MemoryMarshal.Read{T}"/> / <see cref="MemoryMarshal.Write{T}"/>
/// actually consume. For primitive-sized types (<see cref="bool"/>, <see cref="char"/>,
/// and the underlying of enums) the size matches the CLR primitive width.
/// The blittable (managed layout) width of <typeparamref name="T"/>. This is
/// what <see cref="MemoryMarshal.Read{T}"/> / <see cref="MemoryMarshal.Write{T}"/>
/// actually consume. Equals <see cref="Unsafe.SizeOf{T}"/> in the general case,
/// with fixed-width overrides for <see cref="bool"/>, <see cref="char"/>, and
/// enums so the cache value matches the primitive layout width used by those
/// paths.
/// </summary>
public static readonly int Size;
/// <summary>
/// The unmanaged (interop) width via <see cref="Marshal.SizeOf"/>. The marshal
/// path (<see cref="Marshal.PtrToStructure"/>/<see cref="Marshal.StructureToPtr"/>)
/// reads/writes this many bytes. Exceeds <see cref="Size"/> whenever a struct
/// carries inline unmanaged data that the marshaler expands — inline
/// <c>ByValTStr</c>/<c>ByValArray</c> buffers, <c>bool</c> fields (4 bytes per
/// default Win32 BOOL marshaling vs 1 byte managed), etc. For types that do
/// not go through the marshal path, this field is still populated but unused
/// by MemoryBase.
/// </summary>
/// <remarks>
/// <c>Marshal.SizeOf</c> throws for some reference-containing shapes (e.g. a
/// bare <see cref="string"/>). When that happens, we fall back to
/// <see cref="Size"/> — the fallback path is unreachable from production code
/// because types with a reference field always have
/// <see cref="TypeRequiresMarshal"/> true, so MemoryBase reads this field only
/// when it is known to be populated.
/// </remarks>
public static readonly int MarshalSize;
/// <summary>
/// <see langword="true"/> when <typeparamref name="T"/> cannot be copied through
/// the blittable <see cref="System.Runtime.InteropServices.MemoryMarshal"/> path
@@ -35,9 +54,9 @@ public static class MarshalCache<T>
/// </summary>
/// <remarks>
/// The <see cref="MarshalAsAttribute"/> check inspects only top-level fields; a
/// <see cref="MarshalAsAttribute"/> on a field of a nested struct is not detected.
/// Reference-containing nested structs are still caught, because the reference
/// check propagates through nested value types.
/// <see cref="MarshalAsAttribute"/> on a field of a nested struct is not
/// detected. Reference-containing nested structs are still caught, because the
/// reference check propagates through nested value types.
/// </remarks>
public static readonly bool TypeRequiresMarshal;
@@ -51,7 +70,6 @@ public static class MarshalCache<T>
{
// Marshal.SizeOf<char> reports 1 (ANSI char), but the blittable
// MemoryMarshal path reads/writes a char as a 2-byte UTF-16 code unit.
// Use the managed layout width so Size matches what the reader actually uses.
Size = 2;
}
else if (typeof(T).IsEnum)
@@ -60,10 +78,10 @@ public static class MarshalCache<T>
}
else
{
// The blittable path goes through MemoryMarshal, which uses the CLR managed
// layout. Use Unsafe.SizeOf<T> so Size agrees with that layout —
// Marshal.SizeOf<T> can disagree when a struct contains a `bool` field
// (unmanaged width 4 vs managed width 1).
// The blittable path goes through MemoryMarshal, which uses the CLR
// managed layout. Use Unsafe.SizeOf<T> so Size agrees with that
// layout — Marshal.SizeOf<T> disagrees when a struct contains a
// `bool` field (unmanaged 4 vs managed 1).
Size = Unsafe.SizeOf<T>();
}
@@ -73,5 +91,17 @@ public static class MarshalCache<T>
TypeRequiresMarshal =
hasMarshalAsField || RuntimeHelpers.IsReferenceOrContainsReferences<T>();
// MarshalSize is only consulted when TypeRequiresMarshal is true; for the
// rare case where Marshal.SizeOf refuses a shape (ref-containing structs),
// fall back to the managed size so the field stays populated.
try
{
MarshalSize = Marshal.SizeOf<T>();
}
catch (ArgumentException)
{
MarshalSize = Size;
}
}
}