DirectStorage
TurboBench
DirectStorage | TurboBench | |
---|---|---|
18 | 10 | |
653 | 310 | |
2.6% | - | |
4.5 | 8.9 | |
4 months ago | 9 months ago | |
C++ | C | |
MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DirectStorage
-
Game Ready & Studio Driver 535.98 FAQ/Discussion
I don't think the GPU decompression optimizations is in this driver. I updated and using the Bulk Loader Demo test I'm actually seeing lower throughput than before. I used to hit around 24-25GB a sec, now I'm only hitting like 21GB a sec. Although it's possible but I doubt it, it could be related to the Windows 11 March update reducing SSD bandwidth. I haven't run the test in months so it might be related.
-
Pigz: Parallel gzip for modern multi-processor, multi-core machines
The data is compressed with GDeflate, not deflate. The single stream is designed to use the parallelism of a GPU. It is described here:
https://github.com/microsoft/DirectStorage/blob/main/GDeflat...
The GPU decompression benchmark I linked earlier allows you to specify a single file that it will compress with GDeflate (and zlib for comparison). The numbers presented in the docs that come with the benchmark and presented elsewhere are consistent with my own runs using a source file that is highly compressible.
Part of the trick of achieving this speedup is to read the data fast enough. I don't know of any NVMe drive that can reach full speed with a queue depth of 1. While running the benchmark in a windows VM with a GPU passed through, on the linux host I observed that the average read size was about 512k and the queue depth was sometimes over 30.
-
From Project Management to Data Compression Innovator: Building LZ4, ZStandard, and Finite State Entropy Encoder
We already have GDeflate, with permissive sources available for both CPU compression/decompression and GPU decompression in the DirectStorage GitHub repo. I haven't personally played with it yet, but I'll be implementing it in a project I'm working on in the next few months and am pretty excited to do so.
-
[Digital Foundry] The Last of Us Part 1 PC vs PS5 - A Disappointing Port With Big Problems To Address
Wrong: https://github.com/microsoft/DirectStorage/blob/main/Docs/diagrams.mmd
-
DirectStorage Performance Compared: AMD vs Intel vs Nvidia
The github repo has some command line parameters.
- DirectStorage in Star Citizen after Gen12
-
Samsung 990 Pro tested with DirectStorage. The Samsung 990 Pro, like the Sabrent Rocket 4 Plus-G and WD SN850X, has gaming / DirectStorage optimizations.
Only the final test where the results of several SSDs are displayed in the graph is a synthetic one. The first two both support DirectStorage and are designed with Microsoft's recommendations for DirectStorage in mind. That is, random reads of 32k or greater block sizes with high queue depths. This is because you need a high queue depth to be able to saturate NVMe drives.
-
Valve Halves Steam Deck SSD Bandwidth on Some Models
For most it'll be a background element they're not aware of, if you're running up to date win10 onwards you have DS capabilities, you can get the sample from microsoft build it and run it fine.
- DirectStorage API for Windows
-
Looks like PS5 exclusive Returnal is headed to PC
Direct Storage github Samples: https://github.com/microsoft/DirectStorage
TurboBench
-
Zstd Content-Encoding planned to ship with Chrome 123
I'm still unconvinced about this addition. And I don't even dislike Zstandard.
The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2].
According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead. In fact, Meta employs the main author of Zstandard so it would have been easier to tune Zstandard instead of Brotli. While Brotli has some fundamental difference from Zstandard (in particular Brotli doesn't use arithmetic-equivalent coding), no one has concretely demonstrated that difference would prevent Brotli from being fast enough for dynamic contents in my opinion.
[1] https://issues.chromium.org/issues/40196713
[2] https://github.com/powturbo/TurboBench/issues/43
[3] https://docs.google.com/document/d/14dbzMpsYPfkefAJos124uPrl...
- TurboBench: Dynamic/Static web content compression benchmark
-
Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
libdeflate compress better and has faster decompression than igzip.
See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,...
https://github.com/powturbo/TurboBench/issues/4
-
Intel QuickAssist Technology Zstandard Plugin for Zstandard
- https://github.com/powturbo/TurboBench/issues/43
[1] https://github.com/powturbo/TurboBench
-
Variation on RLE to Achieve Lossless Compression for Tabular Data
Compressesing your sample file, we get 823 bytes with brotli
Download TurboBench and make your own tests:
[1] - https://github.com/powturbo/TurboBench
-
Data Compression Drives the Internet. Here’s How It Works
- igzip 1,2 is best for very fast networks > 10MB/s
brotli bring little value at decompression for users
[1] https://github.com/powturbo/TurboBench
[1] https://sites.google.com/site/powturbo/home/web-compression
[2] https://encode.su/threads/2333-TurboBench-Back-to-the-future...
-
Pigz: Parallel gzip for modern multi-processor, multi-core machines
Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors.
[1] https://github.com/powturbo/TurboBench
[2] https://github.com/powturbo/TurboBench/releases
What are some alternatives?
Vortice.Windows - .NET bindings for Direct3D12, Direct3D11, WIC, Direct2D1, XInput, XAudio, X3DAudio, DXC, Direct3D9 and DirectInput.
QAT-ZSTD-Plugin
rapidgzip - Gzip Decompression and Random Access for Modern Multi-Core Machines
DirectX12GameEngine - DirectX 12 .NET game engine
libdeflate - Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
X1nput - Xinput hook for Impulse Trigger emulation
pigz - A parallel implementation of gzip for modern multi-processor, multi-core machines.
display-drivers-uninstaller - Display Driver Uninstaller (DDU) a driver removal utility / cleaner utility
lib842
nvcomp - Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
FPC - FPC - Fast Prefix Coder