sqlite
ffi-overhead
Our great sponsors
sqlite | ffi-overhead | |
---|---|---|
72 | 19 | |
- | 639 | |
- | - | |
- | 0.0 | |
- | 10 months ago | |
C | ||
- | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sqlite
-
Show HN: My Go SQLite driver did poorly on a benchmark, so I fixed it
> I would've probably picked the modernc variation
Heads up about the modernc library, it has been stuck on an old version of sqlite for several months [1]. It seems like maintainer time is the limiting factor [2]. There has been a call to arms on that issue page, the maintainer is looking for help, but it looks like not much has arrived. It seems like it might trace back to blockers in the C-to-Go compiler.
It's a major undertaking and a very impressive piece of work, but I'm not surprised it's a struggle when big roadblocks get hit. I hope they find a way to progress, but I'm very relieved to be seeing some CGo-free alternatives like ncruces/go-sqlite3 emerging. I'm going to give it a try for sure and see if I can live with the compromises.
Squinn-go looks very compelling too, but I don't like that it requires the squinn binary to already be installed on a user's machine, I think that gives with one hand and takes with the other: sure, I get to avoid CGo, but I also lose the turnkey, single-command install, static build benefits Go brings out of the box.
Seconding the point about nitty gritty, I'd read it for sure too!
[1]: https://gitlab.com/cznic/sqlite/-/issues/154
-
Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go
No, but that has the disadvantage of being C compiled into Go, then being compiled into native executable.
I'm actually surprised by how readable this came out; props to the Go->C compiler author. But you can guess that pushing this sort of thing through the Go compiler is going to cause some slowdowns due to sheer paradigm mismatch: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_lin...
-
Show HN: MongoDB Protocol for SQLite
FWIW, we use a version of SQLite transpiled into Go to avoid CGI problems: https://gitlab.com/cznic/sqlite
-
Go port of SQLite without CGo
It could be clearer in the readme, but note that this is a machine translation from C to Go, repeated for every OS-Arch pair. Example of the one you're most likely to use in production: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_linux_amd64.go
-
What would you rewrite in Golang?
Like this? https://gitlab.com/cznic/sqlite
ffi-overhead
-
3 years of fulltime Rust game development, and why we're leaving Rust behind
The overhead for Go in benchmarks is insane in contrast to other languages - https://github.com/dyu/ffi-overhead Are there reasons why Go does not copy what Julia does?
-
Can Fortran survive another 15 years?
What about the other benchmarks on the same site? https://docs.sciml.ai/SciMLBenchmarksOutput/stable/Bio/BCR/ BCR takes about a hundred seconds and is pretty indicative of systems biological models, coming from 1122 ODEs with 24388 terms that describe a stiff chemical reaction network modeling the BCR signaling network from Barua et al. Or the discrete diffusion models https://docs.sciml.ai/SciMLBenchmarksOutput/stable/Jumps/Dif... which are the justification behind the claims in https://www.biorxiv.org/content/10.1101/2022.07.30.502135v1 that the O(1) scaling methods scale better than O(log n) scaling for large enough models? I mean.
> If you use special routines (BLAS/LAPACK, ...), use them everywhere as the respective community does.
It tests with and with BLAS/LAPACK (which isn't always helpful, which of course you'd see from the benchmarks if you read them). One of the key differences of course though is that there are some pure Julia tools like https://github.com/JuliaLinearAlgebra/RecursiveFactorization... which outperform the respective OpenBLAS/MKL equivalent in many scenarios, and that's one noted factor for the performance boost (and is not trivial to wrap into the interface of the other solvers, so it's not done). There are other benchmarks showing that it's not apples to apples and is instead conservative in many cases, for example https://github.com/SciML/SciPyDiffEq.jl#measuring-overhead showing the SciPyDiffEq handling with the Julia JIT optimizations gives a lower overhead than direct SciPy+Numba, so we use the lower overhead numbers in https://docs.sciml.ai/SciMLBenchmarksOutput/stable/MultiLang....
> you must compile/write whole programs in each of the respective languages to enable full compiler/interpreter optimizations
You do realize that a .so has lower overhead to call from a JIT compiled language than from a static compiled language like C because you can optimize away some of the bindings at the runtime right? https://github.com/dyu/ffi-overhead is a measurement of that, and you see LuaJIT and Julia as faster than C and Fortran here. This shouldn't be surprising because it's pretty clear how that works?
I mean yes, someone can always ask for more benchmarks, but now we have a site that's auto updating tons and tons of ODE benchmarks with ODE systems ranging from size 2 to the thousands, with as many things as we can wrap in as many scenarios as we can wrap. And we don't even "win" all of our benchmarks because unlike for you, these benchmarks aren't for winning but for tracking development (somehow for Hacker News folks they ignore the utility part and go straight to language wars...).
If you have a concrete change you think can improve the benchmarks, then please share it at https://github.com/SciML/SciMLBenchmarks.jl. We'll be happy to make and maintain another.
-
When dealing with C, when is Go slow?
If you're calling back and forth between C and Go in a performance critical way. It's one of the slowest languages for wrapping C that there is. I've personally hit this bottleneck in numerous projects, wrapping things like libutp and sqlite. See also https://github.com/dyu/ffi-overhead
-
Understanding N and 1 queries problem
Piling on about overhead (and SQLite), many high-level languages take some hit for using an FFI. So you're still incentivized to avoid tons of SQLite calls.
https://github.com/dyu/ffi-overhead
-
Are there plans to improve concurrency in Rust?
Go doesn't even have native thread stacks. When call any FFI function Go has to switch over to an on-demand stack and coordinate the goroutine and the runtime to avoid preemption and starvation. This is part of why Go's calling overhead is over 30x slower than C/C++/Rust (source). It's understandbly become Go community culture to act like FFI is just not even an option and reinvent everything in Go, but that reinvented Go suffers from these other problems plus many more (such as optimizing far worse than GCC or LLVM).
-
Comparing the C FFI overhead on various languages
Some of the results look outdated. The Dart results look bad (25x slower than C), but looking at the code (https://github.com/dyu/ffi-overhead/tree/master/dart) it appears to be five years old. Dart has a new FFI as of Dart 2.5 (2019): https://medium.com/dartlang/announcing-dart-2-5-super-charge... I'm curious how the new FFI would fare in these benchmarks.
-
Would docker be faster if it were written in rust?
In that case, the libcontainer library would be faster if written in most other languages seeing as Go has unfortunate C-calling performance. In this FFI benchmark Rust is on par with C with 1193ms (total benchmarking time), while Go took 37975ms doing the same.
-
Using Windows API in Julia?
Hi there folks! I'm going to call the Windows API as rapidly as possible and will be doing some calculations with the results, and I thought Julia might be perfect for this task as its FFI is impressively fast, and of course, Julia is fast regarding numbers as well :).
What are some alternatives?
chai - Modern embedded SQL database
go - The Go programming language
sqlite - Go SQLite3 driver
krustlet - Kubernetes Rust Kubelet
go-sqlite3 - sqlite3 driver for go using database/sql
glmark2 - glmark2 is an OpenGL 2.0 and ES 2.0 benchmark
sqlparser-rs - Extensible SQL Lexer and Parser for Rust
kutil - Go Utilities
proteus - A simple tool for generating an application's data access layer.
lzbench - lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors
bun - Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one
CheeseShop - Examples of using PyO3 Rust bindings for Python with little to no silliness.