django-silk VS Ray

Those docs are wrong. They still reference syncdb. Use the README at https://github.com/jazzband/django-silk.

If you're using Django as an API, without templates, django debug toolbar is useless because it requires templates. It won't work with DRF endpoints which means you miss out on profiling and SQL tracing. That's where django-silk shines.

https://github.com/ansible-community/ara/pull/358 https://github.com/b12io/orchestra/pull/830 https://github.com/batiste/django-page-cms/pull/210 https://github.com/carpentries/amy/pull/2130 https://github.com/celery/django-celery/pull/612 https://github.com/django-cms/django-cms/pull/7241 https://github.com/django-oscar/django-oscar/pull/3867 https://github.com/esrg-knights/Squire/pull/253https://github.com/Frojd/django-react-templatetags/pull/64 https://github.com/groveco/django-sql-explorer/pull/474 https://github.com/jazzband/django-silk/pull/550 https://github.com/keras-team/keras/pull/16073 https://github.com/ministryofjustice/cla_backend/pull/773 https://github.com/nitely/Spirit/pull/306 https://github.com/python/pythondotorg/pull/1987 https://github.com/rapidpro/rapidpro/pull/1610 https://github.com/ray-project/ray/pull/22396 https://github.com/saltstack/salt/pull/61647 https://github.com/Swiss-Polar-Institute/project-application/pull/483 https://github.com/UEWBot/dipvis/pull/216

Maybe look at https://github.com/jazzband/django-silk

I've wondered whether it's easier to add data analyst stuff to Elixir that Python seems to have, or add features to Python that Erlang (and by extension Elixir) provides out of the box.

By what I can see, if you want multiprocessing on Python in an easier way (let's say running async), you have to use something like ray core[0], then if you want multiple machines you need redis(?). Elixir/Erlang supports this out of the box.

Explorer[1] is an interesting approach, where it uses Rust via Rustler (Elixir library to call Rust code) and uses Polars as its dataframe library. I think Rustler needs to be reworked for this usecase, as it can be slow to return data. I made initial improvements which drastically improves encoding (https://github.com/elixir-nx/explorer/pull/282 and https://github.com/elixir-nx/explorer/pull/286, tldr 20+ seconds down to 3).

[0] https://github.com/ray-project/ray

Dask, Ray(ray.io), or pyspark(if you have a cluster)

https://github.com/ansible-community/ara/pull/358 https://github.com/b12io/orchestra/pull/830 https://github.com/batiste/django-page-cms/pull/210 https://github.com/carpentries/amy/pull/2130 https://github.com/celery/django-celery/pull/612 https://github.com/django-cms/django-cms/pull/7241 https://github.com/django-oscar/django-oscar/pull/3867 https://github.com/esrg-knights/Squire/pull/253https://github.com/Frojd/django-react-templatetags/pull/64 https://github.com/groveco/django-sql-explorer/pull/474 https://github.com/jazzband/django-silk/pull/550 https://github.com/keras-team/keras/pull/16073 https://github.com/ministryofjustice/cla_backend/pull/773 https://github.com/nitely/Spirit/pull/306 https://github.com/python/pythondotorg/pull/1987 https://github.com/rapidpro/rapidpro/pull/1610 https://github.com/ray-project/ray/pull/22396 https://github.com/saltstack/salt/pull/61647 https://github.com/Swiss-Polar-Institute/project-application/pull/483 https://github.com/UEWBot/dipvis/pull/216

I'd really want use Rust+OpenCV instead of Python+OpenCV to process a lot of images (xxxxxx pieces on a central NAS). I would want to do it by also splitting the work over multiple worker nodes for speed. Unfortunately, I've so far not had the time to figure this out... Meanwhile, a Rust API for Ray is being worked on! https://github.com/ray-project/ray/issues/20609

ray.io doesn't support MPI natively. And thus is not "supercomputer" friendly. Blazer runs on MPI which runs across the NUMA (non-unified memory architecture) setup of a supercomputer. The compute interconnect is 100's of times faster than network remoting, which ray.io uses.

Distributed RL algorithms are provided using ray

I had a similar use case and ended up using ray. https://github.com/ray-project/ray

Currently, rllib's "--export-formats" does nothing; I have folders of checkpoints, but no models. Looks like currently the internal export_model function isn't implemented: https://github.com/ray-project/ray/issues/19021

Neat. Congratulations on the launch!

Apart from the fact that it could deploy to both GCP and AWS, what does it do differently than AWS Batch [0]?

When we had a similar problem, we ran jobs on spots with AWS Batch and it worked nicely enough.

Some suggestions (for a later date):

1. Add built-in support for Ray [1] (you'd essentially be then competing with Anyscale, which is a VC funded startup, just to contrast it with another comment on this thread) and dbt [2].

2. Support deploying coin miners (might be good to widen the product's reach; and stand it up against the likes of consensys).

3. Get in front of many cost optimisation consultants out there, like the Duckbill Group.

If I may, where are you building this product from? And how many are on the team?

Thanks.

[0] https://aws.amazon.com/batch/use-cases/

[1] https://ray.io/

[2] https://getdbt.com/