[Wasm] Use a lazy restore frame when returning from tail calls

5223ee5

JSTests/wasm/stress/tail-call-cross-instance-gc.js

+// Test that GC during a cross-instance tail call chain doesn't corrupt the
+// RestoreInstanceCallee thunk frame. The thunk's Callee slot holds the
+// RestoreInstanceCallee singleton and CodeBlock holds a wasmInstance pointer;
+// both must survive GC scanning.
+    function triggerGC() { $vm.gc(); }
+    assert.eq(instPing.exports.start(1), 255);
+    assert.eq(instPing.exports.start(11), 255);
+    for (let i = 0; i < wasmTestLoopCount; i++) {
+        assert.eq(instPing.exports.start(1), 255);
+        assert.eq(instPing.exports.start(11), 255);
+    }

Each WebAssembly instance pins instance-specific data — memory base, memory bounds, and the instance pointer — in dedicated registers. When a tail call crosses instance boundaries (e.g., via return_call_indirect to a different module), the callee's instance clobbers those registers, and if the chain eventually returns to the original non-tail caller, those registers must be restored before it resumes.

This commit replaces the previous compile-time transitive tail call clobbering analysis with a runtime "restore frame" mechanism. When a Wasm tail call crosses instance boundaries for the first time, a 32-byte frame is lazily inserted just above the caller's argument area, capturing the caller's instance pointer and original return address; the rest of the frame is shifted down 32 bytes. A dedicated thunk (_wasm_restore_frame_return) reloads instance and memory registers on return. The patch removes callCanClobberInstance/computeTransitiveTailCalls and their inlining restriction, and adds ARM64E PAC re-signing plus a JIT cage gate thunk. A reuse check — comparing the current return PC against the thunk address — prevents restore-frame accumulation on repeated cross-instance hops, which the spec requires.

After (runtime restore frame, lazy insertion):
  Caller frame (inst A)
    arg area
    [32-byte restore frame inserted above args, first cross-inst call only]
      slot 0: original CallerFrameAndPC
      slot 1: saved instance A
      slot 2: RestoreFrameCallee
    callee-saves  <- entire frame shifted down 32 bytes

  return -> new cfr -> _wasm_restore_frame_return -> reload inst A regs -> jump to original retPC

  Reuse check: if retPC == thunk addr, skip insertion (no accumulation)

Significance

Cross-instance tail call chains can now inline freely, but ABI correctness now depends on a new runtime frame type that the GC, stack unwinder, sampling profiler, and three Wasm backends (BBQ, OMG, IPInt) must all handle consistently — including ARM64E PAC re-signing of the saved return PC. The dedicated GC stress test signals that the authors knew correctness here was a non-trivial risk.

Audit directions

Stack frame shifting copy loop. The restore frame is inserted by physically shifting the caller's entire frame (callee-saves included) down 32 bytes. The copy loop in emitRestoreInstanceFrameIfNeeded uses a do-while when stack-located sources exist and skips it via a frameSize parameter when not needed. Off-by-one in the copy count or direction could silently corrupt callee-save registers or overwrite adjacent data without triggering a crash immediately.
ARM64E PAC re-signing. The original return PC must be resigned from the caller frame's PAC signing context to the restore frame's PAC context. The commit explicitly handles !Options::allowNonSPTagging() as a special case. A mistake in which context or key is used, or a missed edge case in the non-SP-tagging path, could produce a forgeable return address or a PAC bypass.
Reuse-check correctness across backends. The "no accumulation" invariant depends entirely on the ReturnPC == thunk-address check. This must agree across BBQ, OMG, IPInt, and the JIT cage gate thunk. If a legitimate return address collides with the thunk address, or the gate thunk's address differs from what the check compares against, the restore frame is either skipped (wrong instance restored) or inserted redundantly (spec-violating stack growth).
GC and unwinder correctness. The restore frame is a new frame type. The GC must scan its Callee (RestoreInstanceCallee singleton) and CodeBlock (wasmInstance pointer) slots; the stack unwinder and sampling profiler must recognize and traverse it. Failure in any of these — even intermittent under GC pressure — can cause use-after-free or incorrect profiler symbolication.
JIT cage gate thunk consistency. The wasmRestoreFrame gate thunk must correctly untag JIT-key-signed return PCs. Any divergence from _wasm_restore_frame_return's logic is a security boundary inconsistency that could be leveraged to confuse the untagging.