[JSC] Inline small sorting in DFG / FTL

Component: JSC | cab7a45

JSC's tiers run LLInt (bytecode interpreter) → DFG (speculative JIT) → FTL (B3-backed top tier). DFG speculatively assumes array element types and emits guards; when a guard fails, OSR exit deoptimizes back to the interpreter. Array.prototype.sort previously called a C++ runtime sort entirely outside the JIT.

JSTests/stress/array-sort-inline-boolean-comparator.js

+// Before the fix the intrinsic did only `cmp < 0`, which gave false for boolean
+// `false` (since 0 < 0 is false), so a `() => false` comparator was a no-op and
+// the legacy `(a, b) => a > b` pattern never sorted. The fix routes `cmp === false`
+// through an extra branch that also goes to the shift target.
+
+function sortIt(a, c) { return a.sort(c); }
+
+// Warm up with Int32 comparator so DFG inlines the intrinsic on the hot path.
+for (let i = 0; i < testLoopCount; ++i)
+    sortIt([5, 3, 1, 4, 2], (a, b) => a - b);
+
+const cmp = (a, b) => {
+    if ((a ^ b) & 1) return a > b;  // boolean path
+    return a - b;                   // int32 path
+};

The sort is now emitted inline for Contiguous/Int32/Undecided arrays of ≤16 elements, using insertion sort against a 16-element scratch buffer. Two new DFG nodes carry it: ArraySortCompact extracts and normalizes elements into scratch, falling back to the slow path for holes, Double arrays, or oversized inputs, and ArraySortCommit writes the sorted scratch back. The hard part is OSR exit — the JIT cannot resume at an arbitrary sort iteration if a speculation fails inside the user comparator, so a new LLInt trampoline (array_sort_comparator_return_trampoline) adjusts the return PC and restarts the entire op_call(sort) through the slow path. That is valid per spec since comparator call order is unspecified.

Normal path:
  array.sort(cmp)
       │
  ┌────▼────────────────────┐
  │   ArraySortCompact       │  type-check; copy ≤16 elems to scratch
  │                          │──► slow-path call if: >16, Double, or holey
  └────┬────────────────────┘
       │ scratch[16]
  ┌────▼────────────────────┐
  │  Inline insertion sort   │
  │    inlined cmp() call    │◄─── user JS runs here, can mutate array
  └────┬────────────────────┘
       │ sorted scratch
  ┌────▼────────────────────┐
  │   ArraySortCommit        │  write sorted scratch → original array
  └─────────────────────────┘

OSR-exit path (speculation failure inside inlined cmp()):
  cmp() executes in JIT
       │ guard fails
  array_sort_comparator_return_trampoline  (new LLInt thunk)
       │ adjust return PC
  restart op_call(Array.prototype.sort)    (entire sort, slow path)
       │
  slow-path sort runs on array that cmp() may have already mutated

Significance

This is one of the largest new speculative JIT code paths to land in JavaScriptCore recently: new DFG nodes, a new LLInt trampoline, new type-speculation guards, and comparator inlining interacting at once. The structural fact that matters for security is that arbitrary user JS — the comparator — runs between two JIT-controlled memory operations, ArraySortCompact and ArraySortCommit.

Audit directions

Narrow: comparator side effects between compact and commit. The comparator can grow or shrink the array, transition its structure (Contiguous → Dictionary), trigger GC, or write elements directly, and ArraySortCommit then overwrites the original indexed storage with scratch contents. Check whether commit revalidates that the butterfly and structure have not changed since compact. The tell in any JIT diff is a snapshot node and a writeback node separated by a call that can reach user code.

Wider: OSR-exit state after partial mutation. When exit fires mid-sort inside the comparator, the trampoline restarts from op_call, but the comparator may already have run several times and mutated the array. The slow-path sort then operates on that mutated array; the scratch buffer's lifetime, and whether any part of it was written back before the exit, are the specific things to establish. This generalizes to every JIT operation that restarts rather than resumes — audit whichever other intrinsics use restart-on-exit for the same question of what the aborted attempt already committed.

Also worth sweeping: the boolean-comparator dispatch. The test file documents that the initial implementation treated boolean false as 0 rather than as a shift signal, and the fix adds a CompareStrictEq branch for it. Other comparator return types — NaN, objects with valueOf, boxed doubles — travel different branches; check whether any is mishandled the same way. Separately, the new InlineCallFrame call modes for the sort-comparator context need handling anywhere call stacks are reconstructed (debugger, Error.stack, structured stack traces); missing cases produce wrong output, or wrong behavior if the mode is used for dispatch.