internetarchive VS sanic

https://github.com/jjjake/internetarchive/commit/952ace47e0e...

Me too, first commit was a bit more than 11 years ago.

The --checksum switch of ia verifies the hashes.

first, i've tried python and internetarchive scripts only on XP/Vista with the corresponding version for those OS, without success. I moved to linux, instead. While I have a Raspberry Pi (RPi), I tried first on a Virtual Machine, under Windows. I chose Debian (that's what I run on the RPi) but also had a go at FreeBSD. Both have packages (binaries) ready to go and worked flawlessly. From your post, you have enough skills to set up a virtual machine and install a mainstream linux distro, which is basically downloading an iso, mounting it on the VM, clicking next,next,next,ok,done. You then would boot into the desktop and open the CLI (command line interface). Installing internet archive and python is just a matter of copy pasting a couple of commands. On linux, the internet archive package is https://packages.debian.org/stable/utils/internetarchive and I find it easier than grabbing the binaries through cURL, setting up permissions and whatnot. same for python3. it'll do it's thing (grabs all the files it needs, installs, cleans, all automated, and when it's done you're back at the prompt ($ <-- you asked what this operator means in Python but I think you mean when it shows on the documentation; it's just a command prompt, like it would be on windows cmd, for example c:\archives\uploads> waiting for a command) and ready to throw commands. you first need to setup with your credentials. just ia configure it'll ask all it needs and you're ready to upload stuff. mass uploading different items s basically entering the same command for as many times as it's needed. ia does this for you, using a CSV file -- this involves a bit of pre-processing but when set and done it'll save you a lot of time and wait.

More specifically i'm using ia internetarchive, and Putty 0.75 to log into the Pi. All is updated and outside a screen session works fine. When transfering files I get a progress bar, %, speed and timestamps. But when on a screen all I get it the name of the file being uploaded and nothing else. It only changes when one file finishes and moves to the next or when all is uploaded. No other progress info.

requests - A simple, yet elegant, HTTP library. sanic - Next generation Python web server/framework | Build fast. Run fast. click - Python composable command line interface toolkit elasticsearch-dsl-py - High level Python client for Elasticsearch panel - A high-level app and dashboarding solution for Python internetarchive - A Python and Command-Line Interface to Archive.org coconut - Simple, elegant, Pythonic functional programming

You can do that with the Internet Archive's Python client by jjjake: https://github.com/jjjake/internetarchive

If you're comfortable with command line, you can use the internet archive python script to download stuff from archive.org ( https://github.com/jjjake/internetarchive )

My app fits into multiple categories 1) Since you can search for stages with Atlas Search 2) The entire app is real-time 3) The backend was built with Python and Sanic ASGI framework.

Sanic is very very popular with 16.6k stars, 1.5k forks, opencollective sponsors and a very active github. Falcon is more popular than japronto with 8.9k stars, 898 forks, opencollective sponsors and a very active github too. Despite Japronto been keeped as first place by TechEmPower, Falcon is a way better solution in general with performance similar to fastify an very fast node.js framework that hits 575k requests per second in this benchmark.

> trying to build a lifeboat for Twitter, Python works, but then modules require builds that break.

> Alternatively, any good resources for the above?

There are many, _unbelievably many_ writeups and tools for Python building and packaging. Some of them are really neat! But paralysis of choice is real. So is the reality that many of the new/fully integrated/cutting edge tools, however superior they may be, just won't get long term support to catch on and stay relevant.

When getting started with Python, I very personally like to choose from a few simple options (others are likely to pipe up with their own, and that's great; mine aren't The One Right Way, just some fairly cold/mainstream takes).

1. First pick what stack you'll be using to develop and test software. In Python this is sadly often going to be different from the stack you'll use to deploy/run it in production, but here we are. There are two sub-choices to be made here:

1.a. How will you be running the _python interpreter_ in dev/test? "I just want to use the Python that came with my laptop" is fine to a point, but breaks down a lot sooner than folks expect (again, the reasons for this are variously reasonable and stupid, but here we are). Personally, I like pyenv (https://github.com/pyenv/pyenv) here. It's a simple tool that builds interpreters on your system and provides shell aliases to adjust pathing so they can optionally be used. At the opposite extreme from pyenv, some folks choose Python-in-Docker here (pros: reproducible, makes deployment environments very consistent with dev; cons: IDE/quick build-and-run automations get tricker). There are some other tools that wrap/automate the same stuff that pyenv does.

1.b. How will you be isolating your project's dependencies? "I want to install dependencies globally" breaks down (or worse, breaks your laptop!) pretty quickly, yes it's a bummer. There are three options here: if you really eschew automations/wrappers/thick tools in general, you can do this yourself (i.e. via "pip install --local", optionally in a dedicated development workstation user account); you can use venv (https://docs.python.org/3/library/venv.html stdlib version of virtualenv, yes the names suck and confusing, here we are etc. etc.), which is widely standardized upon and manually use "pip install" while inside your virtualenv, and you can optionally integrate your virtualenv with pyenv so "inside your virtualenv" is easy to achieve via pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv); or you can say "hell with this, I want maximum convenience via a wrapper that manages my whole project" and use Poetry (https://python-poetry.org/). There's no right point on that spectrum, it's up to you to decide where you fall on the "I want an integrated experience and to start prototyping quickly" versus "I want to reduce customizations/wrappers/tooling layers" spectrum.

2. Then, pick how you'll be developing said software: what frameworks or tools you'll be using. A Twitter lifeboat sounds like a webapp, so you'll likely want a web framework. Python has a spectrum of those of varying "thickness"/batteries-included-ness. At the minimum of thickness are tools like Flask (https://flask.palletsprojects.com/en/2.2.x/) and Sanic (like Flask, but with a bias towards performance at the cost of using async and some newer Python programming techniques which tend, in Python, to be harder than the traditional Flask approach: https://sanic.dev). At the maximum of thickness are things like Django/Pyramid. With the minimally-thick frameworks you'll end up plugging together other libraries for things like e.g. database access or web content serving/templating, with the maximally-thick approach that is included but opinionated. Same as before: no right answers, but be clear on the axis (or axes) along with you're choosing.

3. Choose how you'll be deploying/running the software, maybe after prototyping for awhile. This isn't "lock yourself into a cloud provider/hosting platform", but rather a choice about what tools you use with the hosting environment. Docker is pretty uncontentious here, if you want a generic way to run your Python app on many environments. So is "configure Linux instances to run equivalent Python/package versions to your dev/test environment". If you choose the latter, be aware that (and this is very important/often not discussed) many tools that the Python community suggests for local development or testing are very unsuitable for managing production environments (e.g. a tool based around shell state mutation is going to be extremely inconvenient to productionize).

Yeah, that's a lot of choices, but in general there are some pretty obvious/uncontentious paths there. Pyenv-for-interpreters/Poetry-for-packaging-and-project-management/Flask-for-web-serving/Docker-for-production is not going to surprise anyone or break any assumptions. Docker/raw-venv/Django is going to be just as easy to Google your way through.

Again, no one obvious right way (ha!) but plenty of valid options!

Not sure if that's what you were after. If you want a "just show me how to get started"-type writeup rather than an overview on the choices involved, I'm sure folks here or some quick googling will turn up many!

You might want to consider a different name though, as there's already a very popular python web framework called sanic: https://sanic.dev/

For those interested in the nitty-gritty of the application itself, the frontend is developed using vue.js with bulma.io css framework for styling. The backend is running on a laptop in my basement and is served by sanic.dev both for static content and api/websockets.

requests - A simple, yet elegant, HTTP library. sanic - Next generation Python web server/framework | Build fast. Run fast. click - Python composable command line interface toolkit elasticsearch-dsl-py - High level Python client for Elasticsearch panel - A high-level app and dashboarding solution for Python internetarchive - A Python and Command-Line Interface to Archive.org coconut - Simple, elegant, Pythonic functional programming

Personally I haven’t used it outside of trying a few very basic things. I’d recommend blacksheep if you want small, performant and low overhead, or sanic which, in my opinion, is the best choice if you do not need all the Django fluff.

check out https://sanic.dev or https://www.djangoproject.com for your backend