fast-sqlite3-inserts
Lua
Our great sponsors
fast-sqlite3-inserts | Lua | |
---|---|---|
11 | 118 | |
363 | 7,974 | |
- | 2.1% | |
0.0 | 8.5 | |
about 1 year ago | 11 days ago | |
Rust | C | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fast-sqlite3-inserts
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.
I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/
the repo link - https://github.com/avinassh/fast-sqlite3-inserts
-
I/O is no longer the bottleneck
I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.
For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.
I published my findings here [1].
[0] - https://github.com/avinassh/fast-sqlite3-inserts
[1] - https://avi.im/blag/2021/fast-sqlite-inserts/
- Ask HN: Which personal projects got you hired?
-
Is there any language that is as similar as possible to Python in syntax, readability, and features, but is statically typed?
I have a side project where I tried to insert one billion rows in SQLite. I was able to insert 100 million rows using Python under 210 seconds. The same thing with PyPy took 120 seconds. I am wondering what kind of speed boost I would get with Cython
- Ask for benchmark. The owner can’t verify a 18% perf gain, could you?
-
Inserting One Billion Rows in SQLite Under A Minute
Measure, measure, measure! There is a PR which made really minor changes, but it got 2x speed boost with CPython version
- Inserting One Billion Rows in SQLite Under a Minute
- Weekly Coders, Hackers & All Tech related thread - 17/07/2021
-
How we achieved write speeds of 1.4 million rows per second
[somewhat related] Recently, I was benchmarking SQLite inserts and I managed to insert 3.3M records per second (100M in 33 ish seconds) on my local machine - https://github.com/avinassh/fast-sqlite3-inserts Ofcourse the comparison is not apples to apples, but sharing here if anyone finds it interesting
Lua
-
5-Step Approach: ProjectSveltos Event Framework for Kubernetes Deployment with Cilium Gateway API
The EventSource uses the Lua language to search for any services with ports set to 80 or 443 in the ‘argocd’ namespace. More examples can be found here.
-
Building a Wordle Clone with Lua! 🕹
If you're new to the 12 in 24 series, I'm learning and building projects with a new programming language every month - this month, it's the Lua scripting language. You can find source code for the projects I build in the official GitHub repository (check it out, this week's folder contains code for both this and two other bonus projects!).
-
Gearing up for Lua
This month, we're talking about Lua. It's not always a first choice when it comes to programming, but I think there's a lot to enjoy about this little language. Heck, I'm a big game development fan myself - I would look into it even if that was the only reason to.
-
Pluto, a Modern Lua Dialect
It’s Portuguese. It’s the same in the Lua codebase [1].
[1]: https://github.com/lua/lua
-
Fluent Bit with ECS: Configuration Tips and Tricks
If we think we need more flexibility for processing records, we can write our own embedded filters using Lua language. Lua is a highly efficient programming language used mainly for embedded scripting.
- A Linguagem Lua completa 30 anos!
-
The Top 20 Programming Languages and Their Origins
Lua
-
Lua C headers, MacOS
➜ ~ brew info lua ==> lua: stable 5.4.6 (bottled) Powerful, lightweight programming language https://www.lua.org/ /opt/homebrew/Cellar/lua/5.4.6 (29 files, 788.7KB) * Poured from bottle using the formulae.brew.sh API on 2023-05-16 at 11:03:06 From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/l/lua.rb License: MIT ==> Caveats You may also want luarocks: brew install luarocks ==> Analytics install: 16,599 (30 days), 56,745 (90 days), 139,027 (365 days) install-on-request: 1,763 (30 days), 6,266 (90 days), 21,105 (365 days) build-error: 0 (30 days)
-
How do you like code documentation inline in the source code vs. as separate guides, or how would you do it?
I think Lua is a good example of doing documentation well. The source code is commented only as much as needed, mainly with brief comments about things that might not be obvious and a small number of longer explanations of how the architecture works (mainly relevant to developers). It also has a super nice feature that's surprisingly rare: each file has a very short line at the top that describes what the file is, so you don't have to guess based on the filename alone. The API is documented in a single HTML file on the website that has both the high level descriptions of the language and architecture, as well as documentation for each public-facing function. The docs are maintained by hand, but the API is mostly stable, so the docs don't need to change very often.
-
Total Noob With a Question.
This is using the Lua language and the Solar2d game framework
What are some alternatives?
tsbs - Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
julia - The Julia Programming Language
assemblyscript - A TypeScript-like language for WebAssembly.
plum - Multiple dispatch in Python
NvChad - Blazing fast Neovim config providing solid defaults and a beautiful UI, enhancing your neovim experience.
sqlite_micro_logger_arduino - Fast and Lean Sqlite database logger for Arduino UNO and above
lua-nginx-module - Embed the Power of Lua into NGINX HTTP servers
remixdb - RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.
kotlin-script-examples - Examples of Kotlin Scripts and usages of the Kotlin Scripting API
dynamic-dns - An automated dynamic DNS solution for Docker and DigitalOcean
mal - mal - Make a Lisp