bolt
Snappy
bolt | Snappy | |
---|---|---|
6 | 5 | |
2,463 | 6,004 | |
- | 0.6% | |
0.0 | 5.2 | |
over 1 year ago | 21 days ago | |
C++ | C++ | |
Mozilla Public License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
bolt
-
Show HN: Want something better than k-means? Try BanditPAM
> frown on that sort of dataset
That example was definitely contrived and designed to strongly illustrate the point. I'll counter slightly that non-peaky topologies aren't uncommon, but they're unlikely to look anything that would push KMedoids to a pathological state rather than just a slightly worse state ("worse" assuming that KMeans is the right choice for a given problem).
> worth pointing out .. data reference
Totally agreed. I hope my answer didn't come across as too negative. It's good work, and everyone else was talking about the positives, so I just didn't want to waste too much time echoing again that while getting the other points across.
> bolt reference
https://github.com/dblalock/bolt
They say as much in their paper, but they aren't the first vector quantization library by any stretch. Their contributions are, roughly:
1. If you're careful selecting the right binning strategy then you can cancel out a meaningful amount of discretization error.
2. If you do that, you can afford to choose parameters that fit everything nicely into AVX2 machine words, turning 100s of branching instructions into 1-4 instructions.
3. Doing some real-world tests to show that (1-2) matter.
Last I checked their code wasn't very effective for the places I wanted to apply it, but the paper is pretty solid. I'd replace it with a faster KMeans approximation less likely to crash on big data (maybe even initializing with KMedoids :) ), and if the thing you're quantizing is trainable with some sort of gradient update step then you should do a few optimization passes in the discretized form as well.
- Bolt: Faster matrix and vector operations that run on compressed data
- 10x faster matrix and vector operations
-
[R] Multiplying Matrices Without Multiplying
Code: https://github.com/dblalock/bolt
Snappy
-
Why I enjoy using the Nim programming language at Reddit.
Another example of Nim being really fast is the supersnappy library. This library benchmarks faster than Google’s C or C++ Snappy implementation.
-
Stretch iPhone to Its Limit: 2GiB Stable Diffusion Model Runs Locally on Device
It doesn't destroy performance for the simple reason that nowadays memory access has higher latency than pure compute. If you need to use compute to produce some data to be stored in memory, your overall throughput could very well be faster than without compression.
There have been a large amount of innovation on fast compression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms:
* lz4: https://lz4.github.io/lz4/
* Google's snappy: https://github.com/google/snappy
* Facebook's zstd in fast mode: http://facebook.github.io/zstd/#benchmarks
-
Compression with best ratio and fast decompression
Google released Snappy, which is extremely fast and robust (both at compression and decompression), but it's definitely not nearly as good (in terms of compression ratio). Google mostly uses it for real-time compression, for example of network messages - not for long-term storage.
-
How to store item info?
Just compress it! Of course if you will you ZIP, players will able to just open this zip file and change whatever they want. But you can use less popular compression algorithms which are not supported by default Windows File Explorer. Snappy for example.
- What's the best way to compress strings?
What are some alternatives?
composer - Supercharge Your Model Training
zstd - Zstandard - Fast real-time compression algorithm
halutmatmul - Hashed Lookup Table based Matrix Multiplication (halutmatmul) - Stella Nera accelerator
LZ4 - Extremely Fast Compression algorithm
draco - Draco is a library for compressing and decompressing 3D geometric meshes and point clouds. It is intended to improve the storage and transmission of 3D graphics.
brotli - Brotli compression format
PGM-index - 🏅State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes
ZLib - A massively spiffy yet delicately unobtrusive compression library.
LightGBM - A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
LZMA - (Unofficial) Git mirror of LZMA SDK releases
heavydb - HeavyDB (formerly OmniSciDB)
zlib-ng - zlib replacement with optimizations for "next generation" systems.