typedload
ustore
Our great sponsors
typedload | ustore | |
---|---|---|
5 | 15 | |
252 | 486 | |
- | 3.1% | |
8.0 | 9.6 | |
4 days ago | 8 months ago | |
Python | C++ | |
GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
typedload
-
Show HN: Up to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19
Author of typedload here!
FastAPI relies on (not so fast) pydantic, which is one of the slowest libraries in that category.
Don't expect to find such benchmarks on the pydantic documentation itself, but the competing libraries will have them.
[0] https://ltworf.github.io/typedload/
-
Pydantic vs Protobuf vs Namedtuples vs Dataclasses
I wrote typedload, which is significantly faster than pydantic. Just uses normal dataclasses/attrs/NamedTuple, has a better API and is pure Python!
- Informatica serve a qualcosa?
-
Show HN: Python framework is faster than Golang Fiber
I read all the perftests in the repo. I think they nearly all parse a structure that contains a repetition of the same or similar thing a couple hundred thousand times times and the timing function returns the min and max of 5 attempts. I just picked one example for posting.
Not a Python expert, but could the Pydantic tests be possibly not realistic and/or misleading because they are using kwargs in __init__ [1] to parse the object instead of calling the parse_obj class method [2]? According to some PEPs [3], isn't Python creating a new dictionary for that parameter which would be included in the timing? That would be unfortunate if that accounted for the difference.
Something else I think about is if a performance test doesn't produce a side effect that is checked, a smart compiler or runtime could optimize the whole benchmark away. Or too easy for the CPU to do branch prediction, etc. I think I recall that happening to me in Java in the past, but probably not happened here in Python.
[1] https://github.com/ltworf/typedload/blob/37c72837e0a8fd5f350...
[2] https://docs.pydantic.dev/usage/models/#helper-functions
[3] https://peps.python.org/pep-0692/
ustore
-
Up to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19+
Just to clarify, I meant in other projects, like the UKV.
-
Show HN: Up to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19
Yes, we also constantly think about that! In the document collections of UKV, for example, we have interoperability between JSON, BSON, and MessagePack objects [1]. CSV is another potential option, but text-based formats aren't ideal for large scale transmissions.
One thing people do - use two protocols. That is the case with Apache Arrow Flight RPC = gRPC for tasks, Arrow for data. It is a viable path, but compiling gRPC is a nightmare, and we don't want to integrate it into our other libraries, as we generally compile everything from sources. Seemingly, UJRPC can replace gRPC, and for the payload we can continue using Arrow. We will see :)
[1]: https://github.com/unum-cloud/ukv/blob/main/src/modality_doc...
-
UKV: Replacing MongoDB, Neo4J, and Elastic with a single open-source ACID transactional NoSQL database with Zero-Copy Semantics, replaceable backends, and a vast ecosystem of bindings for C, C++, Python, Java, GoLang
![Map](https://github.com/unum-cloud/ukv/raw/main/assets/charts/Intro.png)
-
Beating OpenAI CLIP with 100x less data and compute
Great point! I would be happy to get more input and brain-storm a good pricing model together, one that is fair both for developers and for users.
We have an source project UKV, that partly overlaps with vector-search: https://github.com/unum-cloud/ukv
Another one - UNSW, is a placeholder for now: https://github.com/unum-cloud/unsw
Both will be soon available on cloud marketplaces, but server-less options are a bit harder to cook. Our Discord is the best place to continue conversation: https://discord.gg/Bbh2bjNhvz
Thank you for advice!
- UKV: Modular Transactional NoSQL DBMS Bringing Zero-Copy Semantics to Storage
- PageRank Algorithm for Graph Databases
-
New to Github.. what are Github apps
Most "apps" are for CI/CD pipelines: testing, coverage reports, reviews automation, deployments, and so on. GitHub is a lot more than Git hosting these days. It comes with a web version of VS Code, GitHub Copilot AI, vulnerability detection tools, discussions, roadmap boards, and so on.
-
Python bindings for RocksDB and LevelDB
Documentation
-
Bullshit Graph Database Performance Benchmarks
I am really stunned by this story. It made me check the MemGraph benchmarks section. Don't get me wrong, it may be 10-100x faster than Neo4J in even the most basic operations. Moreover, given the quality of Neo4J, it is hard not to be that much quicker. Even Postgres and MySQL are better at storing graphs than Neo4J.
---
Disclosure: I have worked on Graph Algorithms, Graph Databases, and Database Engines for years, and we are now preparing a commercial solution based on UKV [1]. I don't know anyone at MemGraph or Neo4J. Never used the first. As for the second, I am not a fan.
---
Aside from licensing, there are 3 primary complaints. I will address them individually, and I am open to a discussion.
A. Using Python for Benchmarks instead of Gatling. I don't entirely agree with this. Python still has the fastest-growing programming community while already being one of the 2 most popular languages. Gatling, however, never heard of it. Choosing between the two, I would pick Python. But neither works if you want to design a High-Performance benchmark for a fast system. Without automatic memory management and expensive runtimes, you can only implement those in C, C++, Rust, or another systems-programming language. We have faced that too many times that the benchmark itself works worse than the system it is trying to evaluate [2].
B. Using hardware from 2010 [3], weird datasets [4]. This shocked me. When I looked at the charts [5] and the benchmarking section, it seemed highly professional and good-looking. I wouldn't expect less from a startup with $20M VC funding. But the devil is in the details. I would have never expected anyone benchmarking a new DBMS to use now 13-year-old CPUs and an unknown dataset. Assuming current developer salaries, hiring people to design a DBMS doesn't make sense if you will be evaluating on a $1000 machine is just financially irresponsible. We buy expensive servers, they cost like sports cars or even apartments in poorer countries. It is hard to maintain, but they are essential to quality work. It is sad to see companies taking such shortcuts. But to be a devil's advocate, there is no 1 graph benchmark or dataset that everyone agrees on. So I imagine people experimenting with multiple real datasets of different sizes or generating them systemically using one of the Random Generator algorithms. In UKV, we have used Twitter data to construct both document and graph collections. In the past, we have also used `ci-patent`, `bio-mouse-gene`, `human-Jung2015-M87102575`, and hundreds of other public datasets from the Network Repository and SNAP [6]. There are datasets of every shape and size, reaching around 1 Billion edges, in case someone is searching for data. For us the next step is the reconstruction of the Web from the 300 TB CommonCrawl dataset [7]. There is no such Graph benchmark in existence, but it is the biggest public dataset we could find.
C. Running query different number of times for various engines. This can be justified, and it is how current benchmarks are done. You are tracking not just the mean execution time but also variability, so if at some point results converge, you abrupt before hitting the expected iterations number to save time.
---
LDBC [8] seems like a good contestant for a potential industry standard, but it needs to be completed. Its "Business Intelligence workload" and "Interactive workload" categories exclude any real "Graph Analytics". Running an All-Pairs-Shortest-Paths algorithm on a large external memory graph could have been a much more interesting integrated benchmark. Similarly, one can make large-scale community detection or personalized recommendations based on Graphs and evaluate the overall cost/performance. It, however, poses another big challenge. Almost all algorithm implementations for those problems are vertex-centric. They scale poorly with large sparse graphs that demand edge-centric algorithms, so a new implementation has to be written from scratch. We will try to allocate more resources towards that in 2023 and invite anyone curious to join.
---
[1]: https://github.com/unum-cloud/ukv
- UKV: Open Binary Interface for NoSQL Database Management
What are some alternatives?
codon - A high-performance, zero-overhead, extensible Python compiler using LLVM
kuzu - Embeddable property graph database management system built for query speed and scalability. Implements Cypher.
pydantic-core - Core validation logic for pydantic written in rust
usearch - Fast Open-Source Search & Clustering engine × for Vectors & 🔜 Strings × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍
peps - Python Enhancement Proposals
Unquery - Command line query tool for JSON files
msgspec - A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML
hash-db - Experimental distributed pseudomultimodel keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support and document storage
koda-validate - Typesafe, Composable Validation
yyjson - The fastest JSON library in C
socketify.py - Bringing Http/Https and WebSockets High Performance servers for PyPy3 and Python3
Apache AGE - Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL.