learning_xslt_with_python VS scaling-to-distributed-crawling

I think a middle ground has been reached, XSLT 3.0 allows you to transform your XML into JSON and back. The XSLT 3.0 (2017) processor is available to non-Java languages from Saxonica, for Python "pip install saxonpy" (Linux only)

If you want to see how to do these XML-to-JSON and JSON-to-XML transforms I have written a little learning repo with a CLI: https://github.com/aleph2c/leaning_xslt

Here is Michael Kay's white paper on Transforming JSON using XSLT 3.0: https://www.saxonica.com/papers/xmlprague-2016mhk.pdf

Once your data is in a JSON format, you could implement your compact-binary-format idea around it.

We published a repository and blog post about distributed crawling in Python. It is a bit more complicated than what we've seen so far. It uses external software (Celery for asynchronous task queue and Redis as the database).

We will start to separate concepts before the project grows. We already have two files: tasks.py and main.py. We will create another two to host crawler-related functions (crawler.py) and database access (repo.py). Please look at the snippet below for the repo file, it is not complete, but you get the idea. There is a GitHub repository with the final content in case you want to check it.