[Wasm] Use a lazy restore frame when returning from tail calls

Component: JSC | 5223ee5

JSTests/wasm/stress/tail-call-cross-instance-gc.js

+// Test that GC during a cross-instance tail call chain doesn't corrupt the
+// RestoreInstanceCallee thunk frame. The thunk's Callee slot holds the
+// RestoreInstanceCallee singleton and CodeBlock holds a wasmInstance pointer;
+// both must survive GC scanning.
+
+    function triggerGC() {
+        // Force a full collection. The thunk frame's Callee (RestoreInstanceCallee)
+        // and CodeBlock (wasmInstance) must be scannable by the GC.
+        $vm.gc();
+    }
+    ...
+    // Single cross-instance hop + GC at base case.
+    assert.eq(instPing.exports.start(1), 255);
+    // Deeper chain — GC happens after several cross-instance tail calls.
+    assert.eq(instPing.exports.start(11), 255);
+    for (let i = 0; i < wasmTestLoopCount; i++) {
+        assert.eq(instPing.exports.start(1), 255);
+        assert.eq(instPing.exports.start(11), 255);
+    }

Each WebAssembly instance pins instance-specific data — memory base, memory bounds, instance pointer — in dedicated registers. A tail call that crosses instance boundaries, for example return_call_indirect into a different module, lets the callee's instance clobber those registers; if the chain eventually returns to the original non-tail caller, they must be restored first. The old scheme solved this at compile time with a transitive analysis (computeTransitiveTailCalls, callCanClobberInstance) that marked functions capable of transitively clobbering the instance so their callers restored registers eagerly after every call — conservative, and it blocked inlining. This commit replaces that with a runtime mechanism: on the first cross-instance tail call, a 32-byte restore frame is lazily inserted just above the caller's argument area, capturing the caller's instance pointer and original return address, with the entire frame below shifted down 32 bytes. A dedicated thunk, _wasm_restore_frame_return, reloads the instance and memory registers on return. The patch also adds ARM64E PAC re-signing and a JIT cage gate thunk, and removes the old analysis along with its inlining restriction.

Before (compile-time analysis):
  Caller (inst A)          Callee (inst B)           Return
  │                             │                        │
  ├─[marked:clobbering]────────►│  return ───────────────►│
  │  restore regs eagerly                     (inlining of B blocked)
  │  after every call

After (runtime restore frame, lazy insertion):
  Caller frame (inst A)
  │  arg area
  │  [32-byte restore frame inserted above args, first cross-inst call only]
  │    slot 0: original CallerFrameAndPC
  │    slot 1: saved instance A
  │    slot 2: RestoreFrameCallee
  │  callee-saves  ← entire frame shifted down 32 bytes
  │
  ├─ return_call ──► Callee (inst B) ──► ... ──► return
  │                                              │
  │                                     new cfr → _wasm_restore_frame_return
  │                                              reload inst A regs
  │                                              jump → original retPC
  │
  Reuse check: if retPC == thunk addr, skip insertion (no accumulation)

Significance

ABI correctness for cross-instance tail calls moves from static analysis into a new runtime frame type that the GC, stack unwinder, sampling profiler, and all three Wasm backends must now handle consistently. In exchange, a conservative inlining barrier disappears from Wasm functions involved in cross-instance tail call chains. The spec-mandated "no accumulation" property now rests on a runtime reuse check — comparing the current ReturnPC against the thunk address — rather than on a compile-time property.

Audit directions

Six surfaces are worth separate passes. Stack frame shifting: the restore frame is inserted by physically shifting the caller's entire frame, callee-saves included, down 32 bytes. The copy loop in emitRestoreInstanceFrameIfNeeded uses a do-while when stack-located sources exist and skips it via a frameSize parameter when not needed; an off-by-one in the copy count or direction could silently corrupt callee-save registers or overwrite adjacent data without an immediate crash.

ARM64E PAC re-signing: the original return PC must be re-signed from the caller frame's PAC signing context into the restore frame's context, with !Options::allowNonSPTagging() handled explicitly as a special case. A mistake in which context or key is used, or a missed edge in the non-SP-tagging path, could produce a forgeable return address or a PAC bypass.

Reuse check correctness: the no-accumulation invariant depends entirely on the ReturnPC == thunk-address check, which must agree across BBQ, OMG, and IPInt and between the assembly thunk and the JIT cage gate thunk. If a legitimate return address collides with the thunk address, or the gate thunk's address differs from what the check compares against, the restore frame is either skipped — wrong instance restored — or inserted redundantly, producing spec-violating stack growth.

GC and unwinder correctness: this is a new frame type. The GC must scan its Callee (the RestoreInstanceCallee singleton) and CodeBlock (wasmInstance pointer) slots, and the stack unwinder and sampling profiler must recognize and traverse it. Failure in any of these — even intermittently under GC pressure — can cause use-after-free or incorrect profiler symbolication; the dedicated tail-call-cross-instance-gc.js stress test suggests the authors considered this the sharpest risk.

IPInt vs. BBQ/OMG ABI consistency: both IPInt (via WasmIPIntGenerator) and the BBQ/OMG JIT implement the restore frame, and any difference in frame layout, slot offsets, or embedded thunk address invalidates the reuse check or makes the thunk reload from the wrong slot. Finally, the JIT cage gate thunk — registered so JIT-compiled code uses a consistent entry — must correctly untag JIT-key-signed return PCs; any divergence from _wasm_restore_frame_return's logic is a security boundary inconsistency that could be leveraged to confuse the untagging. The general pattern to carry forward: any runtime-inserted frame type must be simultaneously understood by every backend, the GC, and the unwinder, and each of those four is an independent place to look.