fast-sqlite3-inserts
haxe
Our great sponsors
fast-sqlite3-inserts | haxe | |
---|---|---|
11 | 82 | |
363 | 5,957 | |
- | 1.0% | |
0.0 | 9.7 | |
about 1 year ago | 3 days ago | |
Rust | Haxe | |
MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fast-sqlite3-inserts
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.
I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/
the repo link - https://github.com/avinassh/fast-sqlite3-inserts
-
I/O is no longer the bottleneck
I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.
For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.
I published my findings here [1].
[0] - https://github.com/avinassh/fast-sqlite3-inserts
[1] - https://avi.im/blag/2021/fast-sqlite-inserts/
- Ask HN: Which personal projects got you hired?
-
Is there any language that is as similar as possible to Python in syntax, readability, and features, but is statically typed?
I have a side project where I tried to insert one billion rows in SQLite. I was able to insert 100 million rows using Python under 210 seconds. The same thing with PyPy took 120 seconds. I am wondering what kind of speed boost I would get with Cython
- Ask for benchmark. The owner can’t verify a 18% perf gain, could you?
-
Inserting One Billion Rows in SQLite Under A Minute
Measure, measure, measure! There is a PR which made really minor changes, but it got 2x speed boost with CPython version
- Inserting One Billion Rows in SQLite Under a Minute
- Weekly Coders, Hackers & All Tech related thread - 17/07/2021
-
How we achieved write speeds of 1.4 million rows per second
[somewhat related] Recently, I was benchmarking SQLite inserts and I managed to insert 3.3M records per second (100M in 33 ish seconds) on my local machine - https://github.com/avinassh/fast-sqlite3-inserts Ofcourse the comparison is not apples to apples, but sharing here if anyone finds it interesting
haxe
-
Wax compiler – a tiny language designed to transpile to other languages
This remineds me of Haxe[1]. I like Wax better because of the Common-Lisp-like syntax.
[1]: https://haxe.org
-
Marimo: Interactive Fluffy Ball
I thought this was a three.js demo but it's actually built with a language called haxe [1]. I've never heard of this language before and looks really cool. Makes me want to play with it!
[1] https://haxe.org/
- Haxe 4.3.4
- Ask HN: Does anyone here use Haxe?
-
Ask HN: What are some unpopular technologies you wish people knew more about?
The Haxe programming language (https://haxe.org/). It's insane how unpopular this is compared to its value.
"Haxe can build cross-platform applications targeting JavaScript, C++, C#, Java, JVM, Python, Lua, PHP, Flash, and allows access to each platform's native capabilities. Haxe has its own VMs (HashLink and NekoVM) but can also run in interpreted mode."
It's mostly popular in game dev circles, and is used by: Nortgard, Dead Cells, Papers Please, ... .
-
One Game, by One Man, on Six Platforms: The Good, the Bad and the Ugly
For those interested in cross platform game development, don't forget https://haxe.org/! The usefulness / popularity ratio is very high on this one :).
- Flash Museum – explore more than 130k flash games and animations
-
Is this idea worth pursuing? (a common grammar interface for various interpreted languages written in C)
Sounds like haxe: https://haxe.org/
-
TC39 Proposal: Types as Comments
I really enjoyed programming in AS3, and https://haxe.org/ was really helpful at the time to make development easier.
-
TIL: "private_constant"
Been tinkering in the Haxe programming language recently. I definitely suggest checking it out, but one thing I liked was private constants. I know other languages have this, but its where I've encountered it most recently.
What are some alternatives?
tsbs - Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
Nim - Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).
julia - The Julia Programming Language
eso-light-attack-weave - This is a macro for the game Elder Scrolls Online
plum - Multiple dispatch in Python
Phaser - Phaser is a fun, free and fast 2D game framework for making HTML5 games for desktop and mobile web browsers, supporting Canvas and WebGL rendering. [Moved to: https://github.com/phaserjs/phaser]
sqlite_micro_logger_arduino - Fast and Lean Sqlite database logger for Arduino UNO and above
fut - Fusion programming language. Transpiling to C, C++, C#, D, Java, JavaScript, Python, Swift, TypeScript and OpenCL C.
remixdb - RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.
Fable: F# |> BABEL - F# to JavaScript, TypeScript, Python, Rust and Dart Compiler
dynamic-dns - An automated dynamic DNS solution for Docker and DigitalOcean
love - LÖVE is an awesome 2D game framework for Lua.