fast-sqlite3-inserts VS Killed by Google

I am experimenting with SQLite, where I try inserting 1B rows in under a minute. The current best is inserting 100M rows at 23s. I cut many corners to get performance, but the tweaks might suit your workload.

I have explained my rationale and approach here - https://avi.im/blag/2021/fast-sqlite-inserts/

the repo link - https://github.com/avinassh/fast-sqlite3-inserts

I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.

For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.

I published my findings here [1].

[0] - https://github.com/avinassh/fast-sqlite3-inserts

[1] - https://avi.im/blag/2021/fast-sqlite-inserts/

I have a side project where I tried to insert one billion rows in SQLite. I was able to insert 100 million rows using Python under 210 seconds. The same thing with PyPy took 120 seconds. I am wondering what kind of speed boost I would get with Cython

Measure, measure, measure! There is a PR which made really minor changes, but it got 2x speed boost with CPython version

[somewhat related] Recently, I was benchmarking SQLite inserts and I managed to insert 3.3M records per second (100M in 33 ish seconds) on my local machine - https://github.com/avinassh/fast-sqlite3-inserts Ofcourse the comparison is not apples to apples, but sharing here if anyone finds it interesting

https://killedbygoogle.com/

Their reputation is deserved. Google domains was killed only last year!

And this isn't even the first time I've been burned by Google's decisions. If you're familiar at all with the Google Graveyard, you'll know that Google has a long history of killing off products and services that people have come to rely on. This has happened to me a number of times, in both a personal and professional capacity, and frankly it's getting old.

Just because Google has a couple of decent services that you're willing to pay for doesn't detract from the fact that most of their products have a worse life expectancy than a victorian child in the 1800s. https://killedbygoogle.com

They ruined every single opportunity to be more than an advertising company since Orkut. With scrapped attempts, starts and lack of intention for most of the 2010s to even during the early half of the Pixel Era, they seemingly haven't learnt to stick to something and iterate on it well.

And the fact that over 50% of their revenues come from search and by extension, advertising.

The fact' that til this day, they still haven't evolved from the "throwing shit at the wall then at the fan" strat which explains how they have fumbled so much so quickly.

Yeah. The Google Graveyard really shows how far this can go.

https://killedbygoogle.com

The punchline is that in addition to hundreds of failed hobby projects, their stock is doing great. Monopoly power is a helluva drug.

To me Gemini is just sort of generic and uninteresting. There has to be hundreds or thousands of products and companies based on the name "Gemini" - "Bard" was at least interesting, different and distinct.

I've no idea about the quality of the product itself, I have never had a reason to use it. It's long past cliché now but I wouldn't get too attached to a Google product that is definitely costing a lot of money but which has no clear pathway to turning a profit. I think they will keep it ticking over until the hype train moves on from Chatbots/LLMs, and then it'll join the Google Graveyard @ https://killedbygoogle.com

We're not talking about reliability, we're talking about Google's penchant for killing established products that people use. https://killedbygoogle.com

The website you're referring to: https://killedbygoogle.com/