FFmpeg 8.0 (Part 3): Failed attempts to use Vulkan for AV1 Encoding & VP9 Decoding

Original blog post published in Rendi

FFmpeg 8.0 adds Vulkan support for AV1 encoding and VP9 decoding. Below is an explanation of all these terms, along with my personal experience using these new features in FFmpeg.

This post is part of a series of posts about the new FFmpeg 8.0 release:

1. FFmpeg 8.0 (Part 1): Using Whisper for Native Video Transcription in FFmpeg

2. FFmpeg 8.0 (Part 2): How to use pad_cuda

Vulkan

Vulkan is a cross-platform, open standard set of APIs that allow programs to use GPU hardware. FFmpeg is built with Vulkan 1.3.

FFmpeg 7.1 and 6.1 already supported Vulkan for H264 and HVEC encoding and decoding, as well as AV1 decoding.

Codecs

• H264 and HVEC (also known as H265) are two common codecs used in the common MP4 container.

• AV1 is an open-source codec that is gaining popularity.

• VP9 is an open source codec owned by Google and used in many Google products (including YouTube)

The new FFmpeg 8.0 Vulkan version supports:

• Hardware-accelerated Vulkan AV1 encoding

• Hardware-accelerated Vulkan VP decoding

• Compute-based Vulkan FFv1 encoding-decoding and ProRes Raw decoding

Hardware-accelerated means that Vulkan utilizes the underlying hardware’s capabilities to process specific video commands, while compute-based means that Vulkan uses compute shaders to perform commands (hardware-agnostic).

To install FFmpeg 8.0 with Vulkan, follow the instructions for “How to install FFmpeg 8”. The setup used is a Windows 11 Lenovo laptop with Nvidia RTX 4060 - Driver version 581.29, CUDA version 13.0

Why is Vulkan interesting?

For developers building video applications that should be supported across different hardware and platforms (for example, FFmpeg’s devs), Vulkan is a way to standardize code so you write it once and it works on Linux, Windows, Nvidia, AMD, Intel, and the rest. You don’t need to write special code for Nvidia’s CUDA or AMD’s VAAPI. Vulkan will help FFmpeg’s maintainers ship cross-platform code faster and more reliably.

If you are a FFmpeg user and want to decode and encode on specialized hardware, you will usually need a different FFmpeg command for each hardware platform (Nvidia CUDA, AMD VA-API, Intel QSV, etc.). With Vulkan, if your platform and hardware support it, you can specify one command and use it across different devices. It is not a large gain, because changing FFmpeg commands for each hardware platform is not a major issue.

What are the limitations of Vulkan?

It is not given that Vulkan will run on your platform. The hardware manufacturer needs to provide the required software drivers for the designated platform to support Vulkan, and you need to ensure the drivers you have installed are correct. For example, Nvidia’s drivers that support Vulkan for Linux and Windows, including their release notes, are here: https://developer.nvidia.com/vulkan-driver

Following are a few issues with Nvidia’s 57X drivers with Vulkan

• https://forums.developer.nvidia.com/t/ffmpeg-vulkan-decoding-doesnt-work-rtx-5060-ti-575-57-08/335569

• https://forums.developer.nvidia.com/t/575-release-feedback-discussion/330513/55

Good GitHub discussion about Vulkan and stability issues with FFmpeg and MPV player.

To check which hardware and drivers support Vulkan, you can use this community database.

Vulkan doesn’t add functionality that does not exist on the underlying hardware. So older hardware will not support the new encoding/decoding. To check which Vulkan features your local system supports, use the vulkaninfo tool found in the Vulkan SDK. Make sure to use Vulkan V1.3, as that’s the one shipped with FFmpeg.

For example, to check for support for AV1 encoding and decoding on Nvidia GPU:

vulkaninfo.exe
....
GPU id : 0 (NVIDIA GeForce RTX 4060 Laptop GPU):
....
VK_KHR_video_decode_av1                       : extension revision 1
VK_KHR_video_encode_av1                       : extension revision 1

Vulkan performance

There should not be a performance gain, and maybe even a small performance degradation, because your encoding and decoding commands will go through another abstraction layer before reaching the hardware.

Vulkan is another API standard for connecting application code to underlying GPU hardware, like VAAPI and others, as xkcd put it:

My experience with FFmpeg and Vulkan

I tried utilizing the new Vulkan AV1 encoding and VP9 decoding, but unfortunately, these features did not work. Below is the full description of what worked and what didn’t.

All our experiments used the Big Buck Bunny video and its 16-second snippet.

FFmpeg Vulkan commands are of the structure:

ffmpeg -init_hw_device “vulkan=vk:1” -hwaccel vulkan -hwaccel_output_format vulkan ....

To find the correct index of the GPU (vk:1), use the following command:

./ffmpeg -init_hw_device “vulkan” -v verbose
...
[Vulkan @ 000001bdec619e00] Supported layers:
[Vulkan @ 000001bdec619e00]     VK_LAYER_NV_optimus
[Vulkan @ 000001bdec619e00]     VK_LAYER_NV_present
[Vulkan @ 000001bdec619e00] GPU listing:
[Vulkan @ 000001bdec619e00]     0: Intel(R) Arc(TM) Graphics (integrated) (0x7d55)
[Vulkan @ 000001bdec619e00]     1: NVIDIA GeForce RTX 4060 Laptop GPU (discrete) (0x28a0)
[Vulkan @ 000001bdec619e00] Device 0 selected: Intel(R) Arc(TM) Graphics (integrated) (0x7d55)
...

Encode AV1 with FFmpeg Vulkan

./ffmpeg -init_hw_device “vulkan=vk:1” -hwaccel vulkan -hwaccel_output_format vulkan -i big_buck_bunny_720p_16sec.mp4 -c:v av1_vulkan output_av1.mkv
...
[vost#0:0/av1_vulkan @ 0000029a719635c0] Non-monotonic DTS; previous: 125, current: 42; changing to 125. This may result in incorrect timestamps in the output file.
[vost#0:0/av1_vulkan @ 0000029a719635c0] Non-monotonic DTS; previous: 125, current: 83; changing to 125. This may result in incorrect timestamps in the output file.
Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[h264 @ 0000029a6c81e540] [vk @ 0000029a755b9e80] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[h264 @ 0000029a6c82d240] hardware accelerator failed to decode picture
Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[h264 @ 0000029a74db53c0] get_buffer() failed
[h264 @ 0000029a74db53c0] thread_get_buffer() failed
[h264 @ 0000029a74db53c0] no frame!
Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[h264 @ 0000029a74db5000] get_buffer() failed
[h264 @ 0000029a74db5000] thread_get_buffer() failed
[h264 @ 0000029a74db5000] no frame!
Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[h264 @ 0000029a74db5780] get_buffer() failed
[h264 @ 0000029a74db5780] thread_get_buffer() failed
[h264 @ 0000029a74db5780] no frame!

The command hangs and doesn’t create the desired output.

I was not able to re-encode the H264 video to AV1 using Vulkan, no matter what I tried. I opened this ticket in FFmpeg Forgejo based on Gyan’s instructions.

Running nvenv to re-encode to AV1 worked fine:

./ffmpeg -i big_buck_bunny_720p.mp4 -c:v av1_nvenc output.mkv

If you’re already using av1_nvenc, just keep using it; there’s no reason to switch to Vulkan.

Decoding AV1 with Vulkan (supported in the previous FFmpeg version) and encoding to h264 with CUDA also worked fine:

./ffmpeg -init_hw_device “vulkan=vk:1” -hwaccel vulkan -i big_buck_bunny_720p_av1.mkv -c:v h264_nvenc -c:a aac output.mp4 
...
[out#0/mp4 @ 000001e789517940] video:147136KiB audio:9423KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.259145% frame=14315 fps=769 q=17.0 Lsize=  156965KiB time=00:09:56.33 bitrate=2156.3kbits/s speed=  32x elapsed=0:00:18.61
[aac @ 000001e78efaa100] Qavg: 541.575

Decode VP9 with FFmpeg Vulkan

“Hardware-accelerated VP9 decoding support nowadays is ubiquitous, as most GPUs and SoCs support it natively. Hardware encoding is present in Intel’s Kaby Lake processors and above.”
https://en.wikipedia.org/wiki/VP9

VP9 Vulkan decoding and re-encoding with CUDA:

./ffmpeg  -init_hw_device “vulkan=vk:1” -hwaccel vulkan -i big_buck_bunny_720p_16sec.webm -c:v h264_nvenc -c:a aac output.mp4
...
[opus @ 0000020467207f00] Error parsing Opus packet header.0 bitrate=2207.6kbits/s speed=18.6x elapsed=0:00:00.51
[out#0/mp4 @ 000002046178d840] video:3846KiB audio:256KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.299111%
frame=  384 fps=0.0 q=15.0 Lsize=    4115KiB time=00:00:15.87 bitrate=2123.3kbits/s speed=23.5x elapsed=0:00:00.67
[aac @ 00000204674ffac0] Qavg: 541.625

VP9 Vulkan decoding and re-encoding with CPU:

./ffmpeg  -init_hw_device “vulkan=vk:1” -hwaccel vulkan -i big_buck_bunny_720p_16sec.webm  -c:v libx264   -c:a aac output.mp4
...
[opus @ 000001fe5ed87200] Error parsing Opus packet header.1 bitrate=1336.3kbits/s speed=9.18x elapsed=0:00:01.02
[out#0/mp4 @ 000001fe5990e040] video:2401KiB audio:256KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.450978%
frame=  384 fps=267 q=-1.0 Lsize=    2670KiB time=00:00:15.91 bitrate=1374.0kbits/s speed=11.1x elapsed=0:00:01.43
...

Both commands above created a malformed output video with artifacts:

Re-encoding VP9 to H264 with CUDA worked well, without creating artifacts:

./ffmpeg -init_hw_device cuda -hwaccel cuda -i big_buck_bunny_720p_16sec.webm -c:v h264_nvenc -c:a aac output.mp4
...
[opus @ 0000021cf6b15780] Error parsing Opus packet header.2 bitrate=2178.9kbits/s speed=19.1x elapsed=0:00:00.50
[out#0/mp4 @ 0000021ce78e8c00] video:3718KiB audio:256KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.308708%
frame=  384 fps=0.0 q=12.0 Lsize=    3987KiB time=00:00:15.87 bitrate=2057.5kbits/s speed=24.4x elapsed=0:00:00.65
[aac @ 0000021cf6bb9240] Qavg: 541.625

If you were using CUDA previously, just continue using that and don’t switch to Vulkan.

About my runtime environment:

My setup is a Windows 11 Lenovo laptop with Nvidia RTX 4060 - Driver version 581.29, CUDA version 13.0

• Based on this link, you need a minimum Nvidia driver 550.23

• Based on the Vulkan community, the minimum Nvidia driver version for Vulkan V1.3 is 527

• I tried downgrading our Nvidia driver to version 577, but saw the same results

FFmpeg Vulkan on WSL

To test FFmpeg 8.0 on WSL, I downloaded the N-121064-g424d844534-linux64-gpl build from FFmpeg-Builds, running it on WSL Ubuntu 24.04

FFmpeg Vulkan did not recognize the Nvidia GPU

./ffmpeg -init_hw_device “vulkan” -v verbose
...
[Vulkan @ 0x55f0b44205c0] Supported layers:
[Vulkan @ 0x55f0b44205c0]       VK_LAYER_MESA_device_select
[Vulkan @ 0x55f0b44205c0]       VK_LAYER_MESA_overlay
[Vulkan @ 0x55f0b44205c0]       VK_LAYER_INTEL_nullhw
[Vulkan @ 0x55f0b44205c0] GPU listing:
[Vulkan @ 0x55f0b44205c0]     0: llvmpipe (LLVM 19.1.1, 256 bits) (software) (0x0)
[Vulkan @ 0x55f0b44205c0] Device 0 selected: llvmpipe (LLVM 19.1.1, 256 bits) (software) (0x0)
...

nvidia-smi did recognize the Nvidia GPU:

Running nvenc on the GPU encoding worked fine:

./ffmpeg -i big_buck_bunny_720p.mp4 -c:v av1_nvenc output.mkv