DFG/FTL: inline small Array.prototype.sort with comparator

cab7a45

JSTests/stress/array-sort-inline-boolean-comparator.js

+function sortIt(a, c) { return a.sort(c); }
+for (let i = 0; i < testLoopCount; ++i)
+    sortIt([5, 3, 1, 4, 2], (a, b) => a - b);
+const cmp = (a, b) => {
+    if ((a ^ b) & 1) return a > b;  // boolean path
+    return a - b;                   // int32 path
+};

WebKit's JIT runs in tiers (LLInt → DFG → FTL) and speculates on element types, deoptimizing via OSR exit when a guard fails. Array.prototype.sort previously called a C++ runtime sort outside the JIT entirely. This commit inlines the sort for Contiguous/Int32/Undecided arrays with ≤16 elements via insertion sort against a 16-element scratch buffer. Two new DFG nodes (ArraySortCompact extracts and normalizes into scratch with fallback for holes/Double/oversized; ArraySortCommit writes the sorted scratch back) bracket the inlined comparator. When OSR exit fires inside the inlined comparator, a new LLInt trampoline (array_sort_comparator_return_trampoline) discards the in-flight sort, adjusts the return PC, and restarts the entire op_call(sort) from the slow path — valid per spec because comparator call order is unspecified.

Normal:
  array.sort(cmp)
    |
    +--> ArraySortCompact   (≤16 elems → scratch)
    +--> inline insertion sort with inlined cmp()
    +--> ArraySortCommit    (scratch → original array)

OSR exit inside cmp():
  cmp() in JIT --[guard fails]--> array_sort_comparator_return_trampoline
                                       |
                                       v
                              restart op_call(sort) on slow path
                              (array may already be partially mutated)

Significance

This is one of the largest new speculative JIT code paths in recent JavaScriptCore history, and the test file already documents that the initial implementation shipped with a boolean-comparator semantic bug — a strong indicator that the dispatch logic is not yet hardened.

Audit directions

Comparator side-effects between Compact and Commit. The comparator is arbitrary JS that runs after ArraySortCompact snapshots elements but before ArraySortCommit writes back. A comparator can grow or shrink the array, transition Contiguous to Dictionary, trigger GC, or modify elements directly. Verify ArraySortCommit validates that the array's butterfly and structure have not changed between the two operations.
OSR-exit trampoline state after partial mutation. The trampoline restarts the sort from op_call, but the comparator may already have mutated the array. The scratch buffer's lifetime and whether it has been partially written back before the exit are worth examining for cases where slow-path sort runs on inconsistent state.
Boolean comparator dispatch mismatch. The test file shows the initial implementation missed the cmp === false shift signal. Other return types (NaN, objects with valueOf, boxed doubles) go through different branches; check whether any are mishandled the same way the boolean case was.
InlineCallFrame new call modes. Added call modes for the sort-comparator inline context need handling in any code that reconstructs call stacks (debugger, Error.stack, structured stack traces). Missing cases produce incorrect output or, if used for dispatch, incorrect behavior.