logos VS hashbrown

This is mighty impressive! I've been trying to get some motivation for the mythical rewrite of the proc macro in Logos, and this might just do it for me :D. I'll have a proper look later today and see if any of your findings have something that can be generalized. Also really surprised to see aarch64 doing better than x86_64 since the latter is what I've been optimizing for!

Rust is a very nice langage for implementing compilers, and has a nice ecosystem for it (logos, rust-peg, lalrpop, astmaker -- this one is mine --, etc...).

rust-langdev has a lot of libraries for building compilers in Rust. Perhaps you could use these to make your implementation easier, and revisit it later if you want to build things from scratch. I'd suggest logos for lexing, LALRPOP / chumsky for parsing, and rust-gc for garbage collection.

Thanks! For compile times you might find the CLI version that Andrew Hickman contributed useful, it's undocumented still mostly I fear but shouldn't be hard to use, see original PR: https://github.com/maciejhirsz/logos/pull/248

For my lexer generation purposes, I tend to use https://github.com/maciejhirsz/logos, as it not only generates an easy to use lazy lexer, but the result is also exceptionally fast!

Hi, I'm using rowan to create a parser and want to print more useful error messages with position in the text/file. I'm using logos (https://crates.io/crates/logos) to generate the lexer. Is there a way to get the starting and ending positions of a SyntaxToken? If not I thought of adding my own wrapper struct around the SyntaxTokens.

Is there a way for a lexer created with the logos crate (https://crates.io/crates/logos) to get the starting and ending positions for the tokens?

I wouldn't recommend Logos at this point. This recent bug is quite nasty and seems easy to hit, and the maintainer is unresponsive. Last commit was half a year ago. At this point I consider Logos abandonware, though it would be great if its development continued, or if it were forked.

logos uses "ridiculously fast".

That's because you're looking at a wrapper around the actual implementation (which lives in an external package). Notice "use hashbrown::hash_map as base;" at the top.

There's far more unsafe there: https://github.com/rust-lang/hashbrown/blob/f2e62124cd947b5e...

You might want to check out how popular ecosystem crates do some of these things. Particularly relevant to you are probably crates providing collections, such as smallvec, hashbrown, or indexmap.

https://github.com/topics/swisstable

rust-lang/hashbrown: https://github.com/rust-lang/hashbrown

CuPy has array but not yet hashmaps, or (GPU) SIMD FWICS?

NumPy does SIMD:

Thanks, great point indeed. I am looking into this https://github.com/rust-lang/hashbrown

The way I think about it -- rather naively, I suppose -- is that I care more about the references cells make to each other than the actual grid of cells displayed on a table. The latter feels more like a "view" of the data than an actual data structure?

This also seems to align with the relative priority of (sorted from highest to lowest): figuring out the order of evaluation, calculating those evaluations, and finally displaying the results of the evaluation

updated std's hashbrown dependency to 0.14 which contains some optimizations

The std hashmap is actually very fast and uses state of the art hashmap design, namely because it's implemented by hashbrown

I believe Rust uses hashbrown as the underlying implementation now. This just calculates the number of buckets based on the number of items requested:

https://github.com/rust-lang/hashbrown/blob/009969a860290849...

Is it really the case that rehashing can guarantee that the number of buckets allocated will be sufficient for any given set of keys? In principle you could fail to rehash in a way that reduces collisions after k attempted rehashings.

For example, since the hashbrown crate is marked with #![no_std], it can be used as a dependency for the standard library.

To implement something that cannot be expressed in safe Rust, or at least cannot be expressed succinctly in safe Rust, like fundamental datastructures. The hashbrown crate contains a lot of unsafe code, but it's such high quality that it's now the backing implementation for std::collections::HashMap.