datasette-app VS python-build-standalone

Hah, Softbank isn't the goal here!

I realized that Datasette is the first project of my entire career where if I was still working on it in 15 years time I wouldn't feel bored yet. There's just SO MUCH scope for interesting applications of the core idea.

As such, I want to work on it for decades. But it's lonely working on it alone (the community around it has been growing and is delightful, but it's not the same as having a full-time team.)

So the question I'm trying to answer is how to make the project financially sustainable in the long-run - not just for myself, but so I can pay for a team to work on it with me.

There are plenty of other examples of open source projects that have turned SaaS hosting into a sustainable business model - WordPress and GitLab are just two of the best examples. It feels like it's a reasonably well-trodden path.

Plus... I want people to be able to use my software. Currently to use Datasette as an individual you either have to "pip" or "brew" install it, or you can try the macOS Electron app - https://datasette.io/desktop - but I want newsrooms to be able to use it to collaborate on data. And most newsrooms aren't well equipped to configure a Linux server.

So I realized that a hosted SaaS version can solve two issues at once: it can help the audience I care about actually benefit from the value of the software so far, and it provides a reasonably realistic path to financial sustainability for the project as a whole.

And yeah, I'd also like to make a ton of money out of it myself too!

It would be nice if his stuff worked better, ironically. The Datasette app for Mac seems to be constantly stuck on loading (yes I have 0.2.2):

https://github.com/simonw/datasette-app/issues/139

Amd his screen capture library can't capture Canvas renderings:

https://simonwillison.net/2022/Mar/10/shot-scraper/

Lost two days at work on that.

Speaking of technology not working as expected.

I'd love to get the desktop app working on Linux and Windows.

I did manage to get a prototype working on Windows, despite having VERY little experience working on that platform: https://github.com/simonw/datasette-app/issues/71

The bit I'm stuck on is how to turn that prototype into an application with an installer that's signed so people can download and run it.

I have trouble answering this question myself, and I created it!

The problem I have is that it can be applied to too many different problems.

I personally have used it for the following (a truncated summary):

- Publishing data online to allow other people to explore it, for example https://scotrail.datasette.io and https://russian-ira-facebook-ads.datasettes.com/

- Building websites, by combining it with custom templates. https://datasette.io and https://www.niche-museums.com and https://til.simonwillison.net are three examples

- Building my own combined search engine over a bunch of different data. https://github-to-sqlite.dogsheep.net is this for my GitHub issues and commits and issue comments across 100+ projects

- Similarly, building a code search engine across multiple repos (partly to demonstrate how far you can go with custom plugins): https://ripgrep.datasette.io

- Any time I have a CSV file I open it in the Datasette Desktop macOS app first to start exploring it: https://datasette.io/desktop

- As a prototyping tool. It's the fastest way I know of to get from some data files (CSV or JSON) to a working JSON API - and a GraphQL API too using this plugin: https://datasette.io/plugins/datasette-graphql

- Messing around with geospatial data - here's a write-up of my favourite experiment with that so far: https://simonwillison.net/2021/Jan/24/drawing-shapes-spatial...

This is a bewilderingly wide array of things! And I keep on finding new problems I can apply it to:

Of course, if all you have is a hammer, everything looks like a nail. But thanks to the plugin system (and the amazing flexibility of SQLite under the good) I can reshape my hammer into all sorts of interesting shapes!

I've been trying to capture some of this at https://datasette.io/for

This is one of my biggest marketing challenges for the project though. If someone asks you for an elevator pitch you need to do better than spending 15 minutes talking through a wide ranging bulleted list!

You can use my Datasette software to explore the database: https://datasette.io/desktop - that's the Mac version but you can run the underlying software on Windows too.

Then you can either run "pip install datasette" and "datasette healthkit.db" or you can install the Datasette Desktop app from https://datasette.io/desktop and use that to open the database file.

You may find my Datasette Desktop Mac application useful: it provides a read-only interface over SQLite and cdn oprn both SQLite files and CSV files: https://datasette.io/desktop

This is really interesting to see. I've been trying to solve a similar problem over the past few weeks - bundling up a Python web application as an installable Desktop app, in my case for https://datasette.io/desktop - so it's really interesting to see how they've approached the problem.

I ended up including a full copy of Python using https://github.com/indygreg/python-build-standalone - it looks like they've bundled Conda.

I wrote up detailed notes on how I solved the Python bundling problem in https://simonwillison.net/2021/Sep/8/datasette-desktop/#how-... and in https://til.simonwillison.net/electron/python-inside-electro...

I've been having a ton of fun building this. The code is all open source at https://github.com/simonw/datasette-app - it's my first time working with Electron and the biggest task was figuring out how to bundle Python inside an Electron app, which I wrote about in detail here: https://til.simonwillison.net/electron/python-inside-electron

It also replaces "just" as a task manager for me which is very pleasant.

The fact that the python plugin uses precompiled Python binaries by default instead of building them from source remove common issues I had with the asdf's python plugin at work with missing dependencies.

Just so you know, I encountered two little quirks that needed a fix:

- [Backspace Key Doesn't work in Python REPL](https://github.com/indygreg/python-build-standalone/blob/mai...)

These builds are an alternative: https://github.com/indygreg/python-build-standalone

Those are what Rye and hatch use.

Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)

Bundling Python isn't too bad if you find the right tools for it.

I really like https://github.com/indygreg/python-build-standalone and https://github.com/indygreg/PyOxidizer

A bundled, built standalone Python can be 16 to 32MB (including the full standard library, which you can strip down to just the bits you use to save size). Not tiny, but probably not worth switching programming languages over.

I'm trying to build a "portable" Python package based on those available from https://github.com/indygreg/python-build-standalone/releases.

I'm a huge fan of https://github.com/indygreg/python-build-standalone which provides Python builds that CAN be moved around and work independently of any other Python installation.

I used that for my own Python+Electron app, which I wrote about here: https://til.simonwillison.net/electron/python-inside-electro...

I used to build my own Pythons that are the same everywhere, now I use indygreg's Python builds. Rye will automatically download and manage Python builds from there. No compiling, no divergence.

One interesting tidbit is that it completely ignores your system Python installations, and instead uses precompiled installations of Python by indygreg from PyOxidizer. This means you don't have to deal with installing Python. It just auto downloads the right builds.

wget https://github.com/indygreg/python-build-standalone/releases/download/20220630/cpython-3.10.5+20220630-x86_64_v3-unknown-linux-gnu-install_only.tar.gz -O - | tar -xz && mv python PortablePython

The lack of Ruby support instantly rings an alarm for me because CPython (on POSIX) also is not relocatable, but is listed as support. Turns pit Hermit is actually using a third-party build script[1] instead of the official one. While the python-build-standalone project is quite awesome and indeed is useful for a lot of things, it has enough quirks I would recommend against any generic package distributor to advertise as Python for general use. This in turn makes me lose most confidence on Hermit, unfortunately.

Be careful if you’re also interested in Hermit. These kinds of things bit you up way down the road when you least expect them to.

[1] https://github.com/indygreg/python-build-standalone

If you really want to be sure, you can download an install_only standalone Python build from https://github.com/indygreg/python-build-standalone/releases and install the libraries with the included pip. Then just tar it again to archive it, and use the included python to run your project. The downloaded wheel you get with pip wheel may depend on the Python version so you just save the wheels you must make sure the Python point version is exactly the same.