Fossilize replay linux что это

Fossilize replay linux что это

Steam for Linux

thats a good issue to bring up

i dont use steam shader precaching. preferring to let my GPU handle it.

Here on Steam client beta when the shaders are compiling, I see 8x fossilize_replay processes, I guess it is corresponding to the number or my cpu cores and it takes 100% of the cpu when working. So no I don’t see any problem here.

Do you have only one fozzilize process working on a multi core cpu ? It must take a long time !

I run on nVidia HW and that driver version (GTX1080, tho), Beta client and I also see fossilize_replay take up as many threads as 6-8 (on my 12-core Ryzen), and when «processing shaders» it can take up all available threads (and still take ‘a while’ to be done).

Not sure what the issue might be, anything else updated beside the drivers?

1.) Are you using just 1 or both Options under Shader Pre-Caching?
—>> I ask, cause i’m only using the top Option called Pre-Caching.

2.) In the Shader Tab, it states this.

— May slightly Increase Disk & Bandwith Usage when Enabled.
—->> Could this be why your experiencing issues?

I think you shouldn’t use the term threads but instead fossilize_replay parallel forks, they are both completely different things

It does not really matter to know if fossilize_replay is a true multithread process, what matters is how much fossilize_replay forks are running on your system and the CPU usage.

If you see 8 forks, all taking the whole 100% of your CPU, it works as intended, it does not really matter to know if one of these process is multithread or singlethreaded, most are probably single threaded because I don’t see a reason to multithread a command line tool supposed to be forked, this would be overkill.

Nor multiple single threaded forks, nor one multithreaded process but not both. It would be counter efficient.

Yes it only uses one, I am also using Steam Beta. I wonder if it is a nvidia issue, intel issue, not sure. It is annoying, takes about 5 minutes or so, but it is not a problem beyond that.

I think you should report this on Steam for Linux github, 5min is a lot indeed, maybe something wrong at determining your cpu cores

I took a screenshot of the shaders compiling on my system for RDR2, it took around 3min but for a game of this size and if it used one process, it would surely have taken 8x more time so around 25min. (largest game yet I have caught to compile shaders that long in 3min, it is quicker for most games because they are smaller)

Источник

Fossilize replay linux что это

This repository is discontinued, see new upstream under ValveSoftware/ for further development.

Fossilize is a simple library for serializing various persistent Vulkan objects which typically end up in hashmaps. CreateInfo structs for these Vulkan objects can be recorded and replayed.

• VkSampler (immutable samplers in set layouts)
• VkDescriptorSetLayout
• VkPipelineLayout
• VkRenderPass
• VkShaderModule
• VkPipeline (compute/graphics)

The goal for this project is to cover some main use cases:

• For internal engine use. Extend the notion of VkPipelineCache to also include these persistent objects, so they can be automatically created in load time rather than manually declaring everything up front. Ideally, this serialized cache could be shipped, and applications can assume all persistent objects are already created.
• Create a Vulkan layer which can capture this cache for repro purposes. A paranoid mode would serialize the cache before every pipeline creation, to deal with crashing drivers.
• Easy way of sending shader compilation repros to conformance. Capture internally or via a Vulkan layer and send it off. Normally, this is very difficult with normal debuggers because they generally rely on capturing frames or similar, which doesn’t work if compilation segfaults the driver. Shader compilation in Vulkan requires a lot of state, which requires sending more complete repro applications.

• Some convenience tools to modify/disassemble/spirv-opt parts of the cache.

• Serialize state in application once, replay on N devices to build up VkPipelineCache objects without having to run application.
• Benchmark a pipeline offline by feeding it fake input.

• GCC 4.8+
• Clang
• MSVC 2013/2015/2017+

If rapidjson is not already bundled in your project, you need to check out the submodules.

otherwise, you can set FOSSILIZE_RAPIDJSON_INCLUDE_PATH if building this library as part of your project. It is also possible to use FOSSILIZE_VULKAN_INCLUDE_PATH to override Vulkan header include paths.

Normally, the CLI tools will be built. These require SPIRV-Tools and SPIRV-Cross submodules to be initialized, however, if you’re only building Fossilize as a library/layer, you can use CMake options -DFOSSILIZE_CLI=OFF and -DFOSSILIZE_TESTS=OFF to disable all those requirements for submodules (assuming you have custom include path for rapidjson). Standalone build:

Link as part of other project:

For Android, you can use the android_build.sh script. It builds the layer for armeabi-v7a and arm64-v8a. See the script for more details.

Overall, a binary format which combines JSON with varint-encoded SPIR-V (light compression).

• Magic «FOSSILIZE0000001» (16 bytes ASCII)
• Size of entire binary (64-bit LE)
• JSON magic «JSON » (8 bytes ASCII)
• JSON size (64-bit LE)
• JSON data (JSON size bytes)
• SPIR-V magic «SPIR-V » (8 bytes ASCII)
• SPIR-V size (64-bit LE)
• Varint-encoded SPIR-V words (SPIR-V size bytes)

64-bit little-endian values are not necessarily aligned to 8 bytes.

The JSON is a simple format which represents the various Vk*CreateInfo structures. pNext is currently not supported. When referring to other VK handle types like pImmutableSamplers in VkDescriptorSetLayout , or VkRenderPass in VkPipeline , a 1-indexed format is used. 0 represents VK_NULL_HANDLE and 1+, represents an array index into the respective array (off-by-one). Data blobs (specialization constant data, SPIR-V) are encoded in base64, but I’ll likely need something smarter to deal with large applications which have half a trillion SPIR-V files. When recording or replaying, a mapping from and to real Vk object handles must be provided by the application so the offset-based indexing scheme can be resolved to real handles.

VkShaderModuleCreateInfo refers to an encoded buffer in the SPIR-V block by codeBinaryOffset and codeBinarySize.

The varint encoding scheme encodes every 32-bit SPIR-V word by encoding 7 bits at a time starting with the LSBs, the MSB bit in an encoded byte is set if another byte needs to be read (7 bit) for the same SPIR-V word. Each SPIR-V word takes from 1 to 5 bytes with this scheme.

Источник

Fossilize replay linux что это

Fossilize is a library and Vulkan layer for serializing various persistent Vulkan objects which typically end up in hashmaps. CreateInfo structs for these Vulkan objects can be recorded and replayed.

• VkSampler (immutable samplers in set layouts)
• VkDescriptorSetLayout
• VkPipelineLayout
• VkRenderPass
• VkShaderModule
• VkPipeline (compute/graphics)

The goal for this project is to cover some main use cases:

• For internal engine use. Extend the notion of VkPipelineCache to also include these persistent objects, so they can be automatically created in load time rather than manually declaring everything up front. Ideally, this serialized cache could be shipped, and applications can assume all persistent objects are already created.
• Create a Vulkan layer which can capture this cache for repro purposes when errors occur before we can create a conventional capture.
• Serialize state in application once, replay on N devices to build up VkPipelineCache objects without having to run application.

• GCC 4.8+
• Clang
• MSVC 2013/2015/2017+

If rapidjson is not already bundled in your project, you need to check out the submodules.

otherwise, you can set FOSSILIZE_RAPIDJSON_INCLUDE_PATH if building this library as part of your project. It is also possible to use FOSSILIZE_VULKAN_INCLUDE_PATH to override Vulkan header include paths.

Normally, the CLI tools will be built. These require SPIRV-Tools and SPIRV-Cross submodules to be initialized, however, if you’re only building Fossilize as a library/layer, you can use CMake options -DFOSSILIZE_CLI=OFF and -DFOSSILIZE_TESTS=OFF to disable all those requirements for submodules (assuming you have custom include path for rapidjson). Standalone build:

Link as part of other project:

For Android, you can use the android_build.sh script. It builds the layer for armeabi-v7a and arm64-v8a. See the script for more details.

Overall, a binary database format which contains deflated JSON or deflated varint-encoded SPIR-V (light compression). The database is a bespoke format with extension «.foz». It is designed to be robust in cases where writes to the database are cut off abrubtly due to external instability issues, which can happen when capturing real applications in a layer which applications might not know about. See fossilize_db.cpp for details on the archive format.

The JSON is a simple format which represents the various Vk*CreateInfo structures. When referring to other VK handle types like pImmutableSamplers in VkDescriptorSetLayout , or VkRenderPass in VkPipeline , a hash is used. 0 represents VK_NULL_HANDLE and anything else represents a key. Small data blobs like specialization constant data are encoded in base64. When recording or replaying, a mapping from and to real Vk object handles must be provided by the application so the key-based indexing scheme can be resolved to real handles.

The varint encoding scheme encodes every 32-bit SPIR-V word by encoding 7 bits at a time starting with the LSBs, the MSB bit in an encoded byte is set if another byte needs to be read (7 bit) for the same SPIR-V word. Each SPIR-V word takes from 1 to 5 bytes with this scheme.

Sample API usage

Vulkan layer capture

Fossilize can also capture Vulkan application through the layer mechanism. The layer name is VK_LAYER_fossilize .

To build, enable FOSSILIZE_VULKAN_LAYER CMake option. This is enabled by default. The layer and JSON is placed in layer/ in the build folder.

By default the layer will serialize to fossilize.$hash.$index.foz in the working directory on vkDestroyDevice . However, due to the nature of some drivers, there might be crashes in-between. For this, there are two other modes.

Fossilize in an implicit layer with an enable_environment mechanism. Set this environment variable to automatically load the Fossilize layer.

On Linux and Android, a SIGSEGV handler is registered on instance creation, and the offending pipeline is serialized to disk in the segfault handler. This is very sketchy for general use since it’s not guaranteed to work and it overrides any application handlers, but it should work well if drivers are crashing on pipeline creation. On Windows, the global SEH handler is overridden instead.

If an access violation is triggered, the serialization thread is flushed. A message box will appear on Windows, notifying user about this, and immediately terminates the process after.

Custom file path for capturing state. The actual path which is written to disk will be $FOSSILIZE_DUMP_PATH.$hash.$index.foz . This is to allow multiple processes and applications to dump concurrently.

By default the layer will serialize to /sdcard/fossilize.json on vkDestroyDevice . However, this path might not be writeable, so you will probably have to override your path to something like /sdcard/Android/data/

/capture.json . Make sure your app has external write permissions if using the default path.

Due to the nature of some drivers, there might be crashes in-between. For this, there are two other modes. Options can be set through setprop .

• setprop debug.fossilize.dump_path /custom/path
• setprop debug.fossilize.dump_sigsegv 1

To force layer to be enabled outside application: setprop debug.vulkan.layers «VK_LAYER_fossilize» . The layer .so needs to be part of the APK for the loader to find the layer.

Use adb logcat -s Fossilize to isolate log messages coming from Fossilize. You should see something like:

if capturing is working correctly.

The CLI currently has 3 tools available. These are found in cli/ after build.

This tool is for taking a capture, and replaying it on any device. Currently, all basic PhysicalDeviceFeatures (not PhysicalDeviceFeatures2 stuff) and extensions will be enabled to make sure a capture will validate properly. robustBufferAccess however is set to whatever the database has used.

This tool serves as the main «repro» tool as well as a pipeline driver cache warming tool. After you have a capture, you should ideally be able to repro crashes using this tool. To make replay faster, use —graphics-pipeline-range [start-index] [end-index] and —compute-pipeline-range [start-index] [end-index] to isolate which pipelines are actually compiled.

This tool merges and appends multiple databases into one database.

This tool can convert the binary Fossilize database to a human readable representation and back to a Fossilize database. This can be used to inspect individual database entries by hand.

NOTE: This tool hasn’t been updated since the change to the new database format. It might not work as intended at the moment.

This tool can disassemble any pipeline into something human readable. Three modes are provided:

• ASM (using SPIRV-Tools)
• Vulkan GLSL (using SPIRV-Cross)
• AMD ISA (using VK_AMD_shader_info if available)

TODO is disassembling more of the other state for quick introspection. Currently only SPIR-V disassembly is provided.

NOTE: This tool hasn’t been updated since the change to the new database format. It might not work as intended at the moment.

Runs spirv-opt over all shader modules in the capture and serializes out an optimized version. Useful to sanity check that an optimized capture can compile on your driver.

Running the CLI apps on Android is also supported. Push the binaries generated to /data/local/tmp , chmod +x them if needed, and use the binaries like regular Linux.

Источник