kbe/whitemagic
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix x64 stub ABI and marshal-path sizing; dedupe RPM readers

Browse Source 
x64 call stub was ABI-broken: fixed 0x20 frame left rsp misaligned at the
inner call (callee entry rsp ≡ 0, ABI requires ≡ 8) and, for 5+ args, wrote
stack args over the return address. Compute frame K ≡ 8 (mod 16), K ≥
0x20 + 8*stackArgs, so the callee sees a 16-aligned stack and stack args land
above the shadow window. Load register args as full 64-bit imm64 (was imm32,
which truncated pointers > 4 GiB). BuildCallStub now takes nuint[]; x86 range-
checks each arg against uint.MaxValue instead of silently truncating.

MarshalCache conflated managed and unmanaged width in one Size field: the
blittable path needs Unsafe.SizeOf<T> (bool = 1) while the marshal path needs
Marshal.SizeOf<T> (inline ByValTStr/ByValArray expand past the managed
pointer). Add MarshalSize; MemoryBase picks per TypeRequiresMarshal at all four
IO sites. Prevents PtrToStructure/StructureToPtr from over-reading/overwriting
the pinned scratch buffer (heap corruption on write).

Extract shared RPM/WPM into RpmHelper: honor partial reads (dead Array.Resize
removed), consistent write-return semantics; InProcessReader now guards
MainModule like ExternalReader.

Tests: x64 frame-alignment property + inline-marshal round-trip added (both
fail against the pre-fix code); existing x64 byte-expectation tests updated to
the new frame. Build clean, 100/100 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
kbe
2026-07-21 21:53:22 +02:00
co-authored by Claude Opus 4.8
parent 6fa12d8667
commit cb437ef9b5
7 changed files with 618 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files
WhiteMagic/MarshalCache.cs
+64 -45
View File
@@ -1,88 +1,107 @@
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
namespace WhiteMagic;
/// <summary>
/// Computes and caches marshal-related metadata for type <typeparamref name="T"/>
/// exactly once. <see cref="MemoryBase.Read{T}"/> and <see cref="MemoryBase.Write{T}"/>
/// branch on these cached flags to decide between blittable <c>Span</c>/<c>MemoryMarshal</c>
/// paths and the fallback marshal path.
/// Caches the widths marshalling decisions for type <typeparamref name="T"/>
/// once, at static-constructor time. <see cref="MemoryBase.Read{T}"/> and
/// <see cref="MemoryBase.Write{T}"/> branch on <see cref="TypeRequiresMarshal"/>
/// and pick the appropriate width from this cache.
/// </summary>
/// <typeparam name="T">The type to cache metadata for.</typeparam>
public static class MarshalCache<T>
{
/// <summary>The unmanaged size of <typeparamref name="T"/> in bytes.</summary>
/// <summary>
/// The blittable (managed layout) width of <typeparamref name="T"/>. This is
/// what <see cref="MemoryMarshal.Read{T}"/> / <see cref="MemoryMarshal.Write{T}"/>
/// actually consume. Equals <see cref="Unsafe.SizeOf{T}"/> in the general case,
/// with fixed-width overrides for <see cref="bool"/>, <see cref="char"/>, and
/// enums so the cache value matches the primitive layout width used by those
/// paths.
/// </summary>
public static readonly int Size;
/// <summary>The unmanaged size of <typeparamref name="T"/> as an unsigned integer.</summary>
public static readonly uint SizeU;
/// <summary>
/// The unmanaged (interop) width via <see cref="Marshal.SizeOf"/>. The marshal
/// path (<see cref="Marshal.PtrToStructure"/>/<see cref="Marshal.StructureToPtr"/>)
/// reads/writes this many bytes. Exceeds <see cref="Size"/> whenever a struct
/// carries inline unmanaged data that the marshaler expands — inline
/// <c>ByValTStr</c>/<c>ByValArray</c> buffers, <c>bool</c> fields (4 bytes per
/// default Win32 BOOL marshaling vs 1 byte managed), etc. For types that do
/// not go through the marshal path, this field is still populated but unused
/// by MemoryBase.
/// </summary>
/// <remarks>
/// <c>Marshal.SizeOf</c> throws for some reference-containing shapes (e.g. a
/// bare <see cref="string"/>). When that happens, we fall back to
/// <see cref="Size"/> — the fallback path is unreachable from production code
/// because types with a reference field always have
/// <see cref="TypeRequiresMarshal"/> true, so MemoryBase reads this field only
/// when it is known to be populated.
/// </remarks>
public static readonly int MarshalSize;
/// <summary>
/// <see langword="true"/> when <typeparamref name="T"/> cannot be copied through the
/// blittable <see cref="System.Runtime.InteropServices.MemoryMarshal"/> path and must
/// use <see cref="Marshal.PtrToStructure"/>/<see cref="Marshal.StructureToPtr"/> instead.
/// This is the case when a top-level field carries <see cref="MarshalAsAttribute"/>, or
/// when <typeparamref name="T"/> contains a managed reference
/// (<see cref="System.Runtime.CompilerServices.RuntimeHelpers.IsReferenceOrContainsReferences{T}"/>).
/// <see langword="true"/> when <typeparamref name="T"/> cannot be copied through
/// the blittable <see cref="System.Runtime.InteropServices.MemoryMarshal"/> path
/// and must fall back to <see cref="Marshal.PtrToStructure"/> /
/// <see cref="Marshal.StructureToPtr"/>. This is the case when a top-level field
/// carries <see cref="MarshalAsAttribute"/>, or when <typeparamref name="T"/>
/// contains a managed reference
/// (<see cref="RuntimeHelpers.IsReferenceOrContainsReferences{T}"/>).
/// </summary>
/// <remarks>
/// The <see cref="MarshalAsAttribute"/> check inspects only top-level fields; a
/// <see cref="MarshalAsAttribute"/> on a field of a nested struct is not detected.
/// Reference-containing nested structs are still caught, because the reference check
/// propagates through nested value types.
/// <see cref="MarshalAsAttribute"/> on a field of a nested struct is not
/// detected. Reference-containing nested structs are still caught, because the
/// reference check propagates through nested value types.
/// </remarks>
public static readonly bool TypeRequiresMarshal;
/// <summary><see langword="true"/> when <typeparamref name="T"/> is <see cref="IntPtr"/>.</summary>
public static readonly bool IsIntPtr;
/// <summary>The underlying type code of <typeparamref name="T"/>.</summary>
public static readonly TypeCode TypeCode;
/// <summary>
/// The effective type that the marshaler uses. For an enum this is the underlying
/// integer type; for all other types it is <typeparamref name="T"/> itself.
/// </summary>
public static readonly Type RealType;
static MarshalCache()
{
TypeCode = Type.GetTypeCode(typeof(T));
if (typeof(T) == typeof(bool))
{
Size = 1;
RealType = typeof(T);
}
else if (typeof(T) == typeof(char))
{
// Marshal.SizeOf(char) is 1 (ANSI), but the blittable path reads/writes a
// char as a 2-byte UTF-16 code unit. Size must match the blittable width.
// Marshal.SizeOf<char> reports 1 (ANSI char), but the blittable
// MemoryMarshal path reads/writes a char as a 2-byte UTF-16 code unit.
Size = 2;
RealType = typeof(T);
}
else if (typeof(T).IsEnum)
{
Type underlying = typeof(T).GetEnumUnderlyingType();
Size = Marshal.SizeOf(underlying);
RealType = underlying;
TypeCode = Type.GetTypeCode(underlying);
Size = Marshal.SizeOf(typeof(T).GetEnumUnderlyingType());
}
else
{
Size = Marshal.SizeOf(typeof(T));
RealType = typeof(T);
// The blittable path goes through MemoryMarshal, which uses the CLR
// managed layout. Use Unsafe.SizeOf<T> so Size agrees with that
// layout — Marshal.SizeOf<T> disagrees when a struct contains a
// `bool` field (unmanaged 4 vs managed 1).
Size = Unsafe.SizeOf<T>();
}
SizeU = (uint)Size;
IsIntPtr = RealType == typeof(IntPtr);
bool hasMarshalAsField =
RealType.GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic)
typeof(T).GetFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic)
.Any(f => f.GetCustomAttributes(typeof(MarshalAsAttribute), true).Length != 0);
TypeRequiresMarshal =
hasMarshalAsField || System.Runtime.CompilerServices.RuntimeHelpers.IsReferenceOrContainsReferences<T>();
hasMarshalAsField || RuntimeHelpers.IsReferenceOrContainsReferences<T>();
// MarshalSize is only consulted when TypeRequiresMarshal is true; for the
// rare case where Marshal.SizeOf refuses a shape (ref-containing structs),
// fall back to the managed size so the field stays populated.
try
{
MarshalSize = Marshal.SizeOf<T>();
}
catch (ArgumentException)
{
MarshalSize = Size;
}
}
}