WebCodecs VPX encoder encoded uninitialized data from 10-bit frames

Feed WebCodecs a 10-bit frame and its encoder ships uninitialized heap as video.

Component: WebCore WebCodecs | 790f56e

A WebCodecs VideoEncoder takes VideoFrames backed by platform CVPixelBuffers and passes them to libwebrtc's VP8/VP9/AV1 encoders as native buffers. The encoders then call ToI420(), which on Cocoa is -[RTCCVPixelBuffer toI420]. That method handles only 8-bit NV12 and 32-bit RGB. For any other format it returned the I420 buffer it had already allocated, unfilled, and reported no error.

LibWebRTCVPXInternalVideoEncoder::encode() now converts every pixel buffer that is not 8-bit NV12 into BT709 8-bit NV12, keeping the frame's range. It also sets the frame's color space to BT709, logs the first conversion once per encoder, and fails the encode when there is no buffer. Previously only RGB input was converted.

Significance

10-bit frames, such as those produced by the VP9 profile 2 decoder, were encoded from an unfilled I420 buffer, putting uninitialized heap bytes into a bitstream that JavaScript can read. This was a potential heap disclosure reachable from web content. The fix is a format allowlist in one caller, and toI420 itself is unchanged.

Audit directions

The root behavior in toI420 remains, so look for other callers of -[RTCCVPixelBuffer toI420] and other native-buffer consumers that could still receive unsupported formats: other encoders, scaling and cropAndScale paths, and H264/HEVC WebCodecs and WebRTC senders. Also check 8-bit NV12 variants that pass the new check but differ in range, subsampling or plane layout. In the new conversion, check handling of odd dimensions, strides, visible rect/crop and conversion failure. Finally, check whether the reported decoder configuration stays consistent with the color space and range that the conversion now overrides.

---