VDPU381 and VDPU383 video decoders are integrated into Rockchip RK3588 and RK3576 SoCs, along with variants like RK3588S and RK3576J. Previously, hardware video decoding depended on the Rockchip BSP, but Collabora has announced Linux mainline support for H.264 (AVC) and H.265 (HEVC) video decoding on RK3588 and RK3576 SoCs.
Key points of the H.265/H.264 video decoder implementation on mainline Linux:
– A 17-patch series introduces decoder support, dt-bindings, and device tree nodes.
– New V4L2 HEVC UAPI controls manage short-term and long-term RPS (Reference Picture Set) explicitly.
– A fix for a non-obvious IOMMU restore issue caused by decoder-embedded IOMMU resets.
– A struct-based register programming model ensures completeness, order, and future multi-core readiness.
New V4L2 UAPI controls for HEVC RPS are necessary for VDPU381 (RK3588) and VDP383 (RK3576) video decoders, unlike other decoders (e.g., VeriSilicon) that can ignore them. An API was needed to pass complete RPS tables from userspace to the kernel. Collabora added support in the Virtual Stateless Decoder (visl) driver that shows ftraces with all control parameters. V4L2 UAPI controls were implemented in GStreamer 1.28 (merged) and FFmpeg (preliminary). The new API also enables compatibility with Vulkan Video Decode.
The IOMMU restore issue is noteworthy. The IOMMU core is embedded in Rockchip decoders, resetting along with the decoder and clearing all previous address mappings. However, the kernel treats the IOMMU mapping as valid after a decoder reset. A patch fixes this issue by restoring cached IOMMU mappings after a decoder reset. This affects other IP blocks in Rockchip SoCs, like the RGA 2D graphics accelerator.
Rockchip decoder registers have default values listed in the datasheet. However, hardware may be inconsistent if the kernel skips writing a register, even with default values. Therefore, it’s safer to write all registers. Collabora engineers realized that the write order matters, as writing values in the wrong sequence can break the decoder. They opted to use a C struct for register programming instead of ad-hoc writel() calls or regmap. See the commit for details.
Support may be merged in Linux 7.1, arriving in a little over three months since Linux 6.19 was released in early February, and Linux 7.0 is expected in early April. This news is encouraging yet frustrating, as we can get a recent processor with a vendor BSP or an older processor with mainline Linux, but not a new processor with mainline Linux.
Collabora will continue working on multi-core support for RK3588, as it has two VDPU381 decoder cores, AV1 support on RK3576, VP9 support on RK3588, and add support for the VDPU346 decoder, a VDPU381 variant on RK356X SoCs (RK3562/RK3566/RK3568). A detailed summary of the changes is available on the Collabora website.