arb
tuning_playbook
arb | tuning_playbook | |
---|---|---|
11 | 16 | |
457 | 25,053 | |
0.9% | 2.2% | |
2.2 | 4.7 | |
about 2 months ago | 15 days ago | |
C | ||
GNU Lesser General Public License v3.0 only | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
arb
-
Patriot Missile Floating point Software Problem lead to deaths 28 Americans
You can instead list your criteria for good number format and look at alternatives with those lenses. Floating point is designed for a good balance between dynamic range and precision, and IEEE 754 binary formats can be seen as a FP standard particularly optimized for numerical calculation.
There are several other FP formats. The most popular one is IEEE 754 minus subnormal numbers, followed by bfloat16, IEEE 754 decimal formats (formerly IEEE 854) and posits. Only first two have good hardware supports. The lack of subnormal number means that `a <=> b` can't be no longer rewritten to `a - b <=> 0` among others but is widely believed to be faster. (I don't fully agree, but it's indeed true for existing contemporary hardwares.) IEEE 754 decimal formats are notable for lack of normalization guarantee. Posits are, in some sense, what IEEE 754 would have been if designed today, and in fact aren't that fundamentally different from IEEE 754 in my opinion.
Fixed-point formats share pros and cons of finitely sized integer numbers and you should have no difficulty to analyze them. In short, they offer a smaller dynamic range compared to FP, but its truncation model is much simpler to reason. In turn you will get a varying precision and out-of-bound issues.
Rational number formats look very promising at the beginning, but they are much harder to implement efficiently. You will need a fast GCD algorithm (not Euclidean) and also have to handle out-of-bound numerators and denumerators. In fact, many rational number formats rely on arbitrary-precision integers precisely for avoiding those issues, and inherit the same set of issues---unbounded memory usage and computational overhead. Approximate rational number formats are much rarer, and I'm only aware of the Inigo Quilez's floating-bar experiment [1] in this space.
[1] https://iquilezles.org/articles/floatingbar/
Interval/ball/affine arithmetics and others are means to automatically approximate an error analysis. They have a good property of being never incorrect, but it is still really easy for them to throw up and give a correct but useless answer like [-inf, inf]. Also they are somewhat awkward in a typical procedural paradigm because comparisons will return a tri-state boolean (true, false, unsure). Nevertheless they are often useful when correctly used. Fredrik Johansson's Arb [2] is a good starting point in my opinion.
[2] https://arblib.org/
Finally you can model a number as a function that returns a successively accurate approximation. This is called the constructive or exact real number, and simultaneously most expensive and most correct. One of the most glaring problems is that an equality is not always decidable, and practical applications tend to have various heuristics to get around this fact. Amazingly enough, Android's built-in calculator is one of the most used applications that use this model [3].
[3] https://dl.acm.org/doi/pdf/10.1145/2911981
- Beyond Automatic Differentiation
-
Cosine Implementation in C
https://github.com/JuliaMath/Bessels.jl/blob/master/src/bess...
Thanks! I love it, so easy to understand and follow.
My favourite work on the subject is Fredrik Johansson's:
https://github.com/fredrik-johansson/arb
Whenever I feel down and without energy I just read something in there
-
Math with Significant Figures
Probably the most popular package for dealing with error propagation and arbitrary precision arithmetic in Python is mpmath, more specifically the mp.iv module. For more serious applications I'd take a look at MPFR and Arb, both in C. And there are tons of ball arithmetic and interval arithmetic libraries in Fortran.
-
Function betrayal
You're in good company too. Using intervals to bound error is the entire idea behind the arb library.
-
What are some best practices in dealing with precision errors in computing?
The error bounds approach is probably what you’re looking for. A better search term for that is “interval arithmetic.” There are many good software packages for interval arithmetic, like Arb.
-
Numeric equality
I do agree with your list, so that is something! I will add, balls are underrated, ditto intervals (nominally more efficient, but on x86 switching rounding modes is 20-30 cycles...)
-
Cutting-edge research on numerical representations?
Ball arithmetic looks interesting. As far as I know, arb is the primary implementation.
- Is there a language which can keep track of the potential epsilon error when doing calculations?
- Beware of Fast-Math
tuning_playbook
-
When Random Numbers Are Too Random: Low Discrepancy Sequences
These are also called quasirandom numbers. Despite games, another use case is for hyperparameter search for neural networks.
https://github.com/google-research/tuning_playbook?tab=readm...
-
Hyperparameter Optimization for LLMs via Scaling Laws
[2] https://github.com/google-research/tuning_playbook
-
Beyond Automatic Differentiation
Batch size can be used for regularisation, but using it for that will limit training performance. From the Google Research Tuning Playbook:
> The batch size governs the training speed and shouldn't be used to directly tune the validation set performance. Often, the ideal batch size will be the largest batch size supported by the available hardware.
> […]
> As long as all hyperparameters are well-tuned (especially the learning rate and regularization hyperparameters) and the number of training steps is sufficient, the same final performance should be attainable using any batch size (see Shallue et al. 2018).
https://github.com/google-research/tuning_playbook#choosing-...
The ideal case is full-batch with tuneable regularisation, just the hardware gets expensive.
-
Modeling methodology
Regarding tuning params, this is an excellent read: https://github.com/google-research/tuning_playbook
- About the hardware
- I asked an AI to create an Asmongold story and then had another AI generate voice. There it is dude
-
Trending ML repos of the week 📈
3️⃣ google-research/tuning_playbook
- AI全靠偷欧美开源的
- Deep learning tuning playbook
What are some alternatives?
Arblib.jl - Thin, efficient wrapper around Arb library (http://arblib.org/)
dadaptation - D-Adaptation for SGD, Adam and AdaGrad
calc - C-style arbitrary precision calculator
nn-zero-to-hero - Neural Networks: Zero to Hero
MultiFloats.jl - Fast, SIMD-accelerated extended-precision arithmetic for Julia
ML-Papers-Explained - Explanation to key concepts in ML
tiny-bignum-c - Small portable multiple-precision unsigned integer arithmetic in C
Open-Assistant - OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.
The-RLIBM-Project - A combined repository for all RLIBM prototypes
nanoGPT - The simplest, fastest repository for training/finetuning medium-sized GPTs.
SuiteSparse - SuiteSparse: a suite of sparse matrix packages by @DrTimothyAldenDavis et al. with native CMake support
From-0-to-Research-Scientist-resources-guide - Detailed and tailored guide for undergraduate students or anybody want to dig deep into the field of AI with solid foundation.