edgesearch VS llvm-project

From https://westurner.github.io/hnlog/#comment-30603322 re https://en.wikipedia-on-ipfs.org/wiki/Database_index :

https://github.com/wilsonzlin/edgesearch :

> * Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps *

> "Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers"

WASM or [C++] to WASM?

TIL about Roaring Bitmaps: /?q=roaring+bitmap https://medium.com/@amit.desai03/roaring-bitmaps-fast-data-s...

https://en.wikipedia-on-ipfs.org/wiki/Database_index

https://news.ycombinator.com/item?id=28021766 re: edgesearch, HTTP/3 (QUIC (UDP))

> Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch

>> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers

I've also tried full-text-search in worker by pre-indexing the content, works very fast even with a JS engine - less than 5ms to make a search in 5MB of text.

It runs out of CPU-time at 6MB of text though.

There's someone that made a WASM for the same thing too, it's definitely faster and can handle a bit more text.

https://github.com/wilsonzlin/edgesearch

As far as I know, libstdc++'s representation has two advantages:

First, it simplifies the implementation of `s.data()`, because you hold a pointer that invariably points to the first character of the data. The pointer-less version needs to do a branch there. Compare libstdc++ [1] to libc++ [2].

[1]: https://github.com/gcc-mirror/gcc/blob/065dddc/libstdc++-v3/...

[2]: https://github.com/llvm/llvm-project/blob/1a96179/libcxx/inc...

Basically libstdc++ is paying an extra 8 bytes of storage, and losing trivial relocatability, in exchange for one fewer branch every time you access the string's characters. I imagine that the performance impact of that extra branch is tiny, and massively confounded in practice by unrelated factors that are clearly on libc++'s side (e.g. libc++'s SSO buffer is 7 bytes bigger, despite libc++'s string object itself being smaller). But it's there.

The second advantage is that libstdc++ already did it that way, and to change it would be an ABI break; so now they're stuck with it. I mean, obviously that's not an "advantage" in the intuitive sense; but it's functionally equivalent to an advantage, in that it's a very strong technical answer to the question "Why doesn't libstdc++ just switch to doing it libc++'s way?"

This Ruby implementation is based on mruby and LLVM and it’s commercial software but cheap.

'Computer Architeture: A Quantitative Apporach" and/or more specific design types (mips, arm, etc) can be found under the Morgan Kaufmann Series in Computer Architeture and Design.

"Getting Started with LLVM Core Libraries: Get to Grips With Llvm Essentials and Use the Core Libraries to Build Advanced Tools "

"The Architecture of Open Source Applications (Volume 1) : LLVM" https://aosabook.org/en/v1/llvm.html

"Tourist Guide to LLVM source code" : https://blog.regehr.org/archives/1453

llvm home page : https://llvm.org/

llvm tutorial : https://llvm.org/docs/tutorial/

llvm reference : https://llvm.org/docs/LangRef.html

learn by examples : C source code to 'llvm' bitcode : https://stackoverflow.com/questions/9148890/how-to-make-clan...

See

https://github.com/llvm/llvm-project/issues/88344

https://fortran-lang.discourse.group/t/flang-new-how-to-forc...

You can never mistake type_declaration with an identifier, otherwise the program will not work. Aside from that constraint, you are free to name them whatever you like, there is no one standard, and each parser has it own naming conventions, unless you are planning to use something like LLVM. If you are interested, you can see examples of naming in different language parsers in the AST Explorer.

> There is one way to make the LLVM JIT compiler more usable, but I fear it’s going to take years to be implemented: being able to cache and reuse compiled queries.

Actually, it's implemented in LLVM for years :) https://github.com/llvm/llvm-project/commit/a98546ebcd2a692e...

> It's true, this was a CVE in Rust and not a CVE in C++, but only because C++ doesn't regard the issue as a problem at all. The problem definitely exists in C++, but it's not acknowledged as a problem, let alone fixed.

Can you find a link that substantiates your claim? You're throwing out some heavy accusations here that don't seem to match reality at all.

Case in point, this was fixed in both major C++ libraries:

https://github.com/gcc-mirror/gcc/commit/ebf6175464768983a2d...

https://github.com/llvm/llvm-project/commit/4f67a909902d8ab9...

So what C++ community refused to regard this as an issue and refused to fix it? Where is your supporting evidence for your claims?

For everyone else looking for the magic in this almost 7k lines monster, look at line 6610 [1].

[1] https://github.com/llvm/llvm-project/blob/8ec28af8eaff5acd0d...