PyTorch Vulkan Backend User Workflow¶

Introduction¶

PyTorch 1.7 supports the ability to run model inference on GPUs that support the Vulkan graphics and compute API. The primary target devices are mobile GPUs on Android devices. The Vulkan backend can also be used on Linux, Mac, and Windows desktop builds to use Vulkan devices like Intel integrated GPUs. This feature is in the prototype stage and is subject to change.

Building PyTorch with Vulkan backend¶

Vulkan backend is not included by default. The main switch to include Vulkan backend is cmake option USE_VULKAN , that can be set by environment variable USE_VULKAN .

To use PyTorch with Vulkan backend, we need to build it from source with additional settings. Checkout the PyTorch source code from GitHub master branch.

Optional usage of vulkan wrapper¶

By default, Vulkan library will be loaded at runtime using the vulkan_wrapper library. If you specify the environment variable USE_VULKAN_WRAPPER=0 libvulkan will be linked directly.

Desktop build¶

Vulkan SDK¶

Download VulkanSDK from https://vulkan.lunarg.com/sdk/home and set environment variable VULKAN_SDK

Unpack VulkanSDK to VULKAN_SDK_ROOT folder, install VulkanSDK following VulkanSDK instructions for your system.

After successful build, open another terminal and verify the version of installed PyTorch.

At the time of writing of this recipe, the version is 1.8.0a0+41237a4. You might be seeing different numbers depending on when you check out the code from master, but it should be greater than 1.7.0.

Android build¶

To build LibTorch for android with Vulkan backend for specified ANDROID_ABI .

To prepare pytorch_android aars that you can use directly in your app:

Model preparation¶

Install torchvision, get the default pretrained float model.

Python script to save pretrained mobilenet_v2 to a file:

PyTorch 1.7 Vulkan backend supports only float 32bit operators. The default model needs additional step that will optimize operators fusing

The result model can be used only on Vulkan backend as it contains specific to the Vulkan backend operators.

Using Vulkan backend in code¶

C++ API¶

at::is_vulkan_available() function tries to initialize Vulkan backend and if Vulkan device is successfully found and context is created — it will return true, false otherwise.

.vulkan() function called on Tensor will copy tensor to Vulkan device, and for operators called with this tensor as input — the operator will run on Vulkan device, and its output will be on the Vulkan device.

.cpu() function called on Vulkan tensor will copy its data to CPU tensor (default)

Operators called with a tensor on a Vulkan device as an input will be executed on a Vulkan device. If an operator is not supported for the Vulkan backend the exception will be thrown.

List of supported operators:

Those operators allow to use torchvision models for image classification on Vulkan backend.

Python API¶

torch.is_vulkan_available() is exposed to Python API.

tensor.to(device=’vulkan’) works as .vulkan() moving tensor to the Vulkan device.

.vulkan() at the moment of writing of this tutorial is not exposed to Python API, but it is planned to be there.

Android Java API¶

For Android API to run model on Vulkan backend we have to specify this during model loading:

In this case, all inputs will be transparently copied from CPU to the Vulkan device, and model will be run on Vulkan device, the output will be copied transparently to CPU.

Building android test app with Vulkan¶

Or if you need only specific abi you can set it as an argument:

Add prepared model mobilenet2-vulkan.pt to test applocation assets:

After successful installation, the application with the name ‘MBQ’ can be launched on the device.

Chromium 79 Has Experimental Vulkan Support And You Should NOT Enable It

The latest version of the Google-controlled free software browser Chromium is released and available in the testing repositories of the more up-to-date GNU/Linux distributions. It has a brand new Vulkan rendering mode available which can be enabled in the «Experiments» settings which can be accessed using the special chrome://flags page. Naturally, the first thing that came to mind was «Well, how does it perform?» so we enabled with the hopes of doing some basic benchmarks. That was huge mistake.

chrome://flags has a new experimental option for Vulkan rendering in Chromium 79.

The first thing which comes to mind when I see a new versions of Chromium and Firefox be installed during a system update is «Let’s benchmark it and see how it performs». That did not go so well with Firefox 71 which can’t even finish the Unity 2018 WebGL benchmark due to problems with WebAssembly. The latest Chromium versions and previous Firefox versions run that benchmark just fine.

I’ll typically take a brief look at chrome://flags where experimental flags which can impact performance one way or the other can be altered before I run benchmarks on Chromium and Chromium-based browsers. It’s always interesting to see how the stock settings compare to tuned settings.. This time there was something new on that page, something interesting: «Vulkan — Use vulkan as the graphics backend». That’s clearly something worthy of a quick spin.

Disaster Ensued [ edit ]

The special chrome://gpu page will tell you if the fancy new Vulkan renderer is enabled or not.

At first blush the shiny new Vulkan renderer appeared to work just fine on a desktop machine with a Ryzen CPU and a AMD GPU. The Linux Kernel’s website kernel.org loaded and looked perfectly normal. All was well:

However, re-sizing that window resulted in a .. minor problem:

Moving the window between monitors (after re-starting it) caused the same kind of problem.

The slight rendering glitch did not improve as a short amount of time passed:

Re-starting the browser and going to chrome://flags to disable Vulkan rendering did no longer provide a very user-friendly or intuitive interface for doing so:

Killing and restarting the browser a few times did make it start and render normally. Vulkan works fine until you either resize the window or move between screens most of the time.

Kernel Ring Buffer Story Regarding Chromium’s Vulkan Rendering [ edit ]

It’s always interesting to see what the Linux Kernel’s ring buffer, which can be viewed by executing the dmesg command, has to say when graphical glitches, errors and other problems occur on a GNU/Linux machine.

dmesg had this very informative message explaining what was happening under the hood:

The kernels ring buffer had a very mysterious message from AMD.

The entire ring buffer was filled with pages upon pages with these very mysterious messages.

Versions And Hardware In Question [ edit ]

Testing the Chromium 79.0.3945.79 «freeworld» bulid on two different machines running kernels 5.4.2 and 5.4.5 and newly released Mesa 19.2.8 produced the same result. One has a Ryzen 1600X and a RX470 GPU and the other has a Ryzen 2600 and a RX570. That rules out any hardware problem. amdgpu driver story is that it’s «GPU fault detected»; it says that a lot when it’s not. Two different perfectly fine graphics cards don’t magically break in the same way at the same time and magically revert to a working state when one particular program is closed; that’s a software problem.

The issue is not window manager dependant. A sneaking suspicion that the best desktop environment Xfce’s window manager xfwm4 was responsible made me try KDE Plasma and it’s very fancy effect-filled kwin window manager. The result was hardly an improvement:

Chromium 79.0.3945.79 with Vulkan rendering running in kwin.

This issue could have been amdgpu specific. It’s not. We tested the same Chromium version and enabled Vulkan on a Intel N4200 powered notebook running kernel 5.3.18. The result was a not at all interesting failure. Chromium would launch and work fine until it’s window was moved or re-sized. Doing so resulted in this sad black square:

Chromium 79.0.3945.79 on a Intel N4200 machine using it’s iGPU for Vulkan rendering.

The kernel ring buffer on the N4200 had zero errors. The i915 kernel module for Intel graphics cards does tend to be less vocal than the amdgpu counterpart. A black square where the interface and a web page should be is a clear indication that something is very wrong.

Not Exactly Ready For Prime-Time [ edit ]

Chromium’s new Vulkan rendering mechanism appears to have some minor issues. You should not enable it, you will likely have a bad day if you do.

You should probably stick with Chromium’s default settings. They are mostly fine and it’s performance is actually worse if you enable things like GPU rasterization, that particular option will more than double GPU memory use and reduce performance.

Some of you will want to go to chrome://flags and enable the Vulkan flag and test it. Remember this when disaster occurs: We tried to warn you and you didn’t listen.

We tried to ask AMD developers about the mysterious messages from their amdgpu kernel module. None were available for comment. We did get this comment from Dolphin Emulator developer Ryan Houdek who happened to be present in the secret graphics developer hangout:

» Fun times, good thing it is experimental 😀 «

Vulkan

What Is Vulkan?

Vulkan is a new-generation graphics and compute open standard API that provides high-efficiency, cross-platform access to modern GPUs. Created and evolved by the Khronos® Group standards consortium, Vulkan satisfies the needs of software developers in fields as varied as game, mobile and workstation application development. Vulkan’s explicit API design enables efficient implementations on platforms that span a wide range of mobile, console, embedded and desktop hardware using the Windows, Linux and Android operating systems.

Why Vulkan?

Vulkan is an open standard API that the entire industry can cooperatively evolve to provide access to the latest GPU technologies. The API provides a multi-threading-centric design to leverage modern multi-core CPUs, and provide access to GPUs via multiple parallel command queues. Some of the latest Vulkan features include ray tracing, bindless resources and shader programming using GLSL or HLSL. Vulkan is not tied to a specific platform and enables developers to write GPU code that is portable to diverse devices and operating systems.

Vulkan at NVIDIA

NVIDIA provides fully conformant Vulkan 1.2 drivers across our products including Geforce and Quadro on Windows and Linux, Shield Android TV, and the range of Jetson embedded processors using Android or Linux. NVIDIA Nsight™ tools enable developers with cutting-edge Vulkan application debugging, profiling and optimization capabilities.

Vulkan is available across NVIDIA’s products

NVIDIA is deeply engaged at Khronos and the Vulkan Working Group to help drive Vulkan’s evolution through developing vendor extensions to enable developers access the very latest GPU technologies, and encouraging these capabilities to be adopted as cross-vendor Vulkan specifications at the Khronos Working Group.

To help developers get up and running quickly with Vulkan, this page contains information on NVIDIA’s Vulkan drivers, links to the full source code of educational samples and helper libraries, links the downloadable NVIDIA Nsight tools to help you debug and optimise your Vulkan code, and articles and presentations to help you learn more about what Vulkan can do for you!

NVIDIA will be regularly adding new materials on this page — so keep an eye open for the latest updates!

Vulkan Certified Driver Information

When running on Turing or Ampere hardware these drivers include the

Samples

NVIDIA provides a growing collection of Vulkan and OpenGL samples and tutorials with full source code available on GitHub. Developers can download the samples that interest them, from educational introductions to basic Vulkan workflows and extensions, through examples of how to render scenes with high geometric complexity and ray tracing,and on to professional application use-cases. The repository also contains utility code that can speed up Vulkan development.

Vulkan Developer Tools

Nsight Aftermath

The NVIDIA Nsight Aftermath SDK is a simple library that can be integrated into a game’s crash reporter to generate GPU «mini-dumps» when a GPU hang or exception occurs.

Nsight Systems

NVIDIA Nsight Systems is a system-wide analysis tool for visualizing CPU / GPU interactions to help eliminate GPU idle, stutter and CPU bottlenecks. It can trace Vulkan on the CPU & GPU, interoperating GPU APIs, and many CPU or OS events.

Nsight Graphics

NVIDIA Nsight Graphics enables developers to debug & profile Vulkan applications. With powerful low-level profiling features such as GPU Trace, it’s easier than ever to identify GPU performance issues and optimize with confidence.

RTX Ray Tracing in Vulkan

In November 2020, Khronos released the final versions of the Vulkan Ray Tracing extensions that seamlessly integrate ray tracing into the existing Vulkan framework. Vulkan is the industry’s first open, cross-vendor, cross-platform standard that can be used to access RTX ray tracing acceleration.

Alongside the release of ray tracing support in the Khronos Vulkan SDK in December 2020, NVIDIA now ships the Vulkan Ray Tracing extensions in our production Vulkan drivers . All RTX GPUs are supported, together with GeForce GTX 1660 with 6GB+ of memory and GeForce GTX 1060+ with 6GB+ of memory.

NVIDIA has also released Quake II RTX version 1.4 that uses Vulkan Ray Tracing to significantly enhance the visual quality of this well-loved classic running with ray-traced lighting, shadows, and reflections — making Quake II RTX the world’s first cross-vendor ray tracing Vulkan application! NVIDIA releases the full source code on GitHub serving as a great example for developers who want to dive into the details of how this remastering was achieved.

Getting Started with Vulkan Ray Tracing

A tutorial on how to use Vulkan Ray Tracing to create a complete mini-path tracer using the final Vulkan Ray Tracing extensions

An updated Vulkan Ray Tracing Tutorial using the final Vulkan Ray Tracing extensions

Read about how the 2020.6 release of the NVIDIA Nsight Graphics tool supports Vulkan Ray Tracing