hpy VS Tile38

There is a merge request up to add autogen rust bindings to hpy

https://github.com/hpyproject/hpy/pull/457

Are you referencing https://github.com/hpyproject/hpy?

I do hope it takes off.

The HPy project [0] seems like a promising way out of this.

[0] https://hpyproject.org/

socketify.py breaks the record for Python no other Python WebFramework/Server as able to reach 6.2 mi requests per second before in TechEmPower Benchmarks, this puts Python at the same level of performance that Golang, Rust and C++ for web development, in fact Golang got 5.2 mi req/s in this same round. Almost every server or web framework tries to use JIT to boost the performance, but only socketify.py deliveries this level of performance, and even without JIT socketify.py is twice as fast any other web framework/server in active development, and still can be much more optimized using HPy (https://hpyproject.org/). Python will get even faster and faster in future!

I think Propose a better messaging for Python is the option and a lot of languages will learn it from Rust, because rust erros are the best described errors I see in my life lol. Cargo is amazing and I think we will need a better poetry/pip for sure, HPy project will modernize extensions and packages 📦 too https://hpyproject.org/

It also show that PyPy3 will not magically boost your performance, you need to integrate in a manner that PyPy3 can optimize and delivery CPU performance, with a more complex example maybe it can help more. But why socketify is so much faster using PyPy3? The answer is CFFI, socketify did not use Cython for integration and cannot delivery the full performance on Python3, this will be solved with HPy.

HPy integration to better support CPython, PyPy and GraalPython

Absolutely everything in CPython is a PyObject, and that can’t be changed without breaking the C API. A PyObject contains (among other things) a type pointer, a reference count, and a data field; none of these things can be changed without (again) breaking the C API.

There have definitely been attempts to modernize; the HPy project (https://hpyproject.org/), for instance, moves towards a handle-oriented API that keeps implementation details private and thus enables certain optimizations.

[2] https://github.com/tidwall/tile38

Experimental format to help readability of a long rant:

1.

According to the OP, there's a "terrifying tale of VACUUM in PostgreSQL," dating back to "a historical artifact that traces its roots back to the Berkeley Postgres project." (1986?)

2.

Maybe the whole idea of "use X, it has been battle-tested for [TIME], is robust, all the bugs have been and keep being fixed," etc., should not really be that attractive or realistic for at least a large subset of projects.

3.

In the case of Postgres, on top of piles of "historic code" and cruft, there's the fact that each user of Postgres installs and runs a huge software artifact with hundreds or even thousands of features and dependencies, of which every particular user may only use a tiny subset.

4.

In Kleppmann's DDOA [1], after explaining why the declarative SQL language is "better," he writes: "in databases, declarative query languages like SQL turned out to be much better than imperative query APIs." I find this footnote to the paragraph a bit ironic: "IMS and CODASYL both used imperative query APIs. Applications typically used COBOL code to iterate over records in the database, one record at a time." So, SQL was better than CODASYL and COBOL in a number of ways... big surprise?

Postgres' own PL/pgSQL [2] is a language that (I imagine) most people would rather NOT use: hence a bunch of alternatives, including PL/v8, on its own a huge mass of additional complexity. SQL is definitely "COBOLESQUE" itself.

5.

Could we come up with something more minimal than SQL and looking less like COBOL? (Hopefully also getting rid of ORMs in the process). Also, I have found inspiring to see some people creating databases for themselves. Perhaps not a bad idea for small applications? For instance, I found BuntDB [3], which the developer seems to be using to run his own business [4]. Also, HYTRADBOI? :-) [5].

6.

A usual objection to use anything other than a stablished relational DB is "creating a database is too difficult for the average programmer." How about debugging PostgreSQL issues, developing new storage engines for it, or even building expertise on how to set up the instances properly and keep it alive and performant? Is that easier?

I personally feel more capable of implementing a small, well-tested, problem-specific, small implementation of a B-Tree than learning how to develop Postgres extensions, become an expert in its configuration and internals, or debug its many issues.

Another common opinion is "SQL is easy to use for non-programmers." But every person that knows SQL had to learn it somehow. I'm 100% confident that anyone able to learn SQL should be able to learn a simple, domain-specific, programming language designed for querying DBs. And how many of these people that are not able to program imperatively would be able to read a SQL EXPLAIN output and fix deficient queries? If they can, that supports even more the idea that they should be able to learn something different than SQL.

----

1: https://dataintensive.net/

2: https://www.postgresql.org/docs/7.3/plpgsql-examples.html

3: https://github.com/tidwall/buntdb

4: https://tile38.com/

5: https://www.hytradboi.com/

I actually worked on a project that did this. We used a database called "Tile38" [1] which used an R-Tree to make geospatial queries speedy. It was pretty good.

Our dataset was ~150 GiB, I think? All in RAM. Took a while to start the server, as it all came off disk. Could have been faster. (It borrowed Redis's query language, and its storage was just "store the commands the recreate the DB, literally", IIRC. Dead simple, but a lot of slack/wasted space there.)

Overall not a bad database. Latency serving out of RAM was, as one should/would expect, very speedy!

[1]: https://tile38.com/

I ported it from Go and use it for my Tile38 project.