[JSC] Fix wasm type parsing regression by making RTT formal canonicalized types

Component: JSC | 103a40a

JSTests/wasm/stress/cross-module-rtt-identity.js

// Two separately compiled modules declare the same recursive struct type.
// Isorecursive canonicalization must map both to the *same* RTT pointer
// so that a ref.cast in module B on a value produced by module A succeeds.
 
const { makeLeaf, leafVal } = new WebAssembly.Instance(new WebAssembly.Module(M_A)).exports;
const { castAndRead }       = new WebAssembly.Instance(new WebAssembly.Module(M_B)).exports;
 
// Cross-module ref.cast. If canonicalization is per-module rather than
// isorecursive-per-recgroup, this cast traps and castAndRead throws.
const b = castAndRead(leaf);
if (b !== v)
    throw new Error(`B castAndRead(${v}): expected ${v}, got ${b}`);

WebAssembly GC adds heap-allocated structs, arrays, and recursive types. Runtime Types (RTTs) carry type identity for those objects and are compared by pointer equality when executing ref.cast and subtype checks — which means correctness demands isorecursive canonicalization: two independently compiled modules declaring structurally identical recursive types must receive the same canonical RTT pointer. The old design maintained two parallel representations, TypeDefinition for parsing and RTT for runtime, creating lifetime complexity plus a performance regression from repeated traversal during GC structure construction. This commit removes TypeDefinition entirely. Temporary Subtype, Projection, and RecursionGroup objects now live in a SegmentedVector inside a new TypeSectionState, surviving only for the duration of type-section parsing, after which everything is baked into pre-allocated RTTs, deduplicated, and registered with the process-wide TypeInformation singleton. RTTs hold RefPtr references to sibling RTTs, intentionally forming refcount cycles for mutually-recursive types that are permanently leaked — matching V8's current approach — pending a future cycle collector.

Before:                                   After:
Type section parse                        Type section parse
  └─ TypeDefinition (heap, refcounted)      └─ TypeSectionState
       FunctionSignature                         RecursionGroup  ┐ SegmentedVector,
       StructType                                Projection      │ freed after parse
       ArrayType                                Subtype         ┘
       RecursionGroup                           │
       Projection              bake into pre-allocated RTT
       Subtype                                  │
  ↓                            TypeInformation::canonicalize()
creatCanonicalRTTForType()       ├─ already registered? → reuse canonical RTT
  └─► RTT (canonical)            └─ new?               → register, expose process-wide

At runtime: TypeDefinition gone. Only RTT exists.
Mutually-recursive RTTs hold RefPtr<RTT> to siblings → intentional refcount cycle (leaked).

Significance

This is a ground-up rewrite of the type infrastructure that every WebAssembly GC operation — struct allocation, array access, ref.cast, subtype checks — depends on. Bugs introduced during a large mechanical refactor of this surface translate directly into type confusion reachable from JavaScript.

Audit directions

Canonicalization hash and equality over recursive types: canonicalizeRecursionGroupImpl and canonicalizeSingletonImpl in WasmTypeDefinition.cpp implement the deduplication that underpins every cross-module ref.cast. hashRTTForRecGroup and equalRTTsForRecGroup operate over recursive structures that use placeholder references during construction; subtle mistakes handling self-references or mutual references can make structurally distinct types compare equal — type confusion — or make equivalent types compare unequal, breaking valid casts in ways that may be exploitable through fallback paths.

TypeSectionState pointer escape: Subtype, Projection, and RecursionGroup objects are allocated in a SegmentedVector inside TypeSectionState and swept when type-section parsing ends. The roughly hundred-file refactor must ensure no pointer to these survives into an RTT, ModuleInformation, or any other longer-lived structure — a single missed raw pointer becomes a UAF once the type section is parsed. This generalizes: every arena-scoped parsing structure in the tree deserves the same escape audit.

RTT cycle anchoring completeness: the commit moves from raw TypeIndex references to RefPtr<const RTT> anchor slots (TypeSlot::rttAnchor, StructFieldEntry::rttAnchor, RTTArrayPayload::m_elementTypeAnchor) to keep sibling RTTs alive through mutual-recursion cycles. Any payload field holding a reference to another RTT without a corresponding anchor RefPtr reintroduces exactly the dangling pointer this commit exists to fix, and the diff size makes one easy to miss.

ref.cast and subtype check correctness: isSubRTT and isStrictSubRTT rely on canonical RTT pointer equality, and the replacement of typeIndexFromFunctionIndexSpace with rttFromFunctionIndexSpace across the JIT backends (BBQ, OMG, IPInt), B3 IR nodes, and DFG/FTL is mechanical but vast. An off-by-one or incorrect RTT selection at any callsite in the cast or call-ref path is a type-confusion primitive.