boost VS simdjson

coost is a cross-platform C++ basic library with both performance and ease of use. It is like boost, but much smaller, the static library built on linux and mac is only about 1MB in size. Although small, it provides enough powerful features:

Not so much ease, as flexibility.

In the end, the thing that matters the most for software is being able to get logic into code as efficiently as possible. This includes being able to write concise code, being able to execute it and see results, debug it efficiently, use libraries easily, and deploy it to production. Python has all of this.

The rest of the stuff, like strong typing, memory safety, e.t.c are at best academic. The supposed advantages of those just don't hold up once you start to look into the real world. Linux, which runs on most devices that support an os hardware wise, is written purely in C. Python is used as a backend for very big projects like Youtube, Instagram, Spotify, e.t.c. Its also used to run Openpilot (https://github.com/commaai/openpilot), which has performance on par with Teslas autopilot.

Meanwhile in Java world, with strict typing, you have egregious vulnerabilities like log4shell, amongst others (https://java-0day.com/).

Language evolution is also a thing to look at with this stuff. The more "strict" you try to make a language, the worse its going to become as people are necessarily going to find hacks around it. With java, type safety strict features like having getters and setters get abstracted away behind an annotation processor that hacks the AST (Lombok), and thats not only considered ok, but is encouraged to be used. With C++, template metaprogramming got extremely out of hand with https://www.boost.org/, where the error messages for one thing used to be pages long. Rust manage to sneak this under the radar with the unsafe clause, which is going to see standard use in many codebases, thus negating any of its advantages.

In the end, good code comes from good developers, full stop. Every codebase will necessarily have tests for production deployment, and anything that language features don't catch during compilation or static checking can be checked with testing if you have developers that understand what they are doing and can write appropriate testing frameworks.

And based on that, its pretty attractive to use Python especially when you consider developer time. And the flexibility means you can write your code in different forms to suit your use case, where it be OOP with MyPy type checking, functional, imperative, or super complex if you want.

For those who didn't know what is Boost, it's a C++ library that helps to prevent re-inventing the wheel while trying to program something quite complex as example looping only with macro, Boost Preprocessor. Fortunately, Boost Preprocessor Repeat also works with plain C, not only C++. So, my OrangePi board can calculate hash at compile-time. Unfortunately, my SIX Hash algorithm requires sizeof(input) and Boost... won't... work... with it. Hours of workarounds, no luck.

If you have the ability to use third-party libraries (though if you can't this project is going to be a nightmare, lol...) I would recommend using the Beast library from the Boost collection of libraries. It's a little bit more verbose than some options, but not that much more, and it's better maintained. REST webservices are built on top of the HTTP framework, so it's just a matter of sending a HTTP GET request to a server (or POST/UPDATE/DELETE, depending on how exactly the api on the other end is implemented) and reading the response you get back. This is a very basic sample of a client sending a GET request to a server. If you need to change this to do a POST (or some other kind of request), there's only two real changes that need to be made:

The person you replied to used slightly confusing terminology. Conda deals with non-python packages. As in if you wanted to install boost for C++.

I'm not sure if its just me but I'm finding I can't access any of the download links on the Boost Website.

rich standard library and Boost;

Boost is also another ubiquitous library. Lots of code that doesn't make it into the standard kind of ends up here. Lots of code that gets into the standard starts here. Boost.Asio might end up being our network API in 23.

It's also supported by simdjson [0] (which has a lot of language bindings [1]):

> Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s

[0] https://simdjson.org/

[0] https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...

String parsing is negligible compared to the speed of the DOM which is glacially slow: https://news.ycombinator.com/item?id=38835920

Come on, people, make an effort to learn how insanely fast computers are, and how insanely inefficient our software is.

String parsing can be done at gigabytes per second: https://github.com/simdjson/simdjson If you think that is the slowest operation in the browser, please find some resources that talk about what is actually happening in the browser?

Thanks for all the detailed information! That answers a bunch of my questions and the implementation of strlen is nice.

The instruction I was thinking of is pshufb. An example ‘weird’ use can be found for detecting white space in simdjson: https://github.com/simdjson/simdjson/blob/24b44309fb52c3e2c5...

This works as follows:

1. Observe that each ascii whitespace character ends with a different nibble.

2. Make some vector of 16 bytes which has the white space character whose final nibble is the index of the byte, or some other character with a different final nibble from the byte (eg first element is space =0x20, next could be eg 0xff but not 0xf1 as that ends in the same nibble as index)

3. For each block where you want to find white space, compute pcmpeqb(pshufb(whitespace, input), input). The rules of pshufb mean (a) non-ascii (ie bit 7 set) characters go to 0 so will compare false, (b) other characters are replaced with an element of whitespace according to their last nibble so will compare equal only if they are that whitespace character.

I’m not sure how easy it would be to do such tricks with vgather.vv. In particular, the length of the input doesn’t matter (could be longer) but the length of white space must be 16 bytes. I’m not sure how the whole vlen stuff interacts with tricks like this where you (a) require certain fixed lengths and (b) may have different lengths for tables and input vectors. (and indeed there might just be better ways, eg you could imagine an operation with a 256-bit register where you permute some vector of bytes by sign-extending the nth bit of the 256-bit register into the result where the input byte is n).

Additionally, if you like low level stuff, check out libfmt (https://github.com/fmtlib/fmt) - not a big project, not difficult to understand. Or something like simdjson (https://github.com/simdjson/simdjson).

Everything you said is totally reasonable. I'm a big fan of napkin math and theoretical upper bounds on performance.

simdjson (https://github.com/simdjson/simdjson) claims to fully parse JSON on the order of 3 GB/sec. Which is faster than OP's Go whitespace parsing! These tests are running on different hardware so it's not apples-to-apples.

The phrase "cannot go faster than this" is just begging for a "well ackshully". Which I hate to do. But the fact that there is an existence proof of Problem A running faster in C++ SIMD than OP's Probably B scalar Go is quite interesting and worth calling out imho. But I admit it doesn't change the rest of the post.

I have same question as /u/JDRiverRun : how do you deal with JSON, do you parse json on Rust side or on Emacs side. I see that you are requiring json.el in your lspce.el, but I haven't looked through entire file carefully. If you parse on Rust side, do you use simdjson (there are at least two Rust bindings to it)? If yes, what are your impressions, experiences compared to more "standard" json library?