Dask VS fastapi-crudrouter

Data scientists work on phenomenally large datasets, and Dask is a handy tool for exploration within the confines of a single cloud VM or their local PCs. Location data visualization is an essential part of deciding further algorithm development and roadmap for projects. This lays the foundation for data engineering and science to work at scale, with petabytes of data.

Dask

We're using a lot of Python. In addition to these, gridMET, Dask, HoloViz, and kerchunk.

I wrote this for speeding up the RPC messaging in dask, but figured it might be useful for others as well. The source is available on github here: https://github.com/jcrist/msgspec.

Dask: Distributed data frames, machine learning and more

To do that, we are efficiently using Dask, simply creating on-demand local (or remote) clusters on task run() method:

Look into Dask, see: https://dask.org/

[1] https://dask.org/

FastAPI does have an equivalent. Something like this? https://github.com/awtkns/fastapi-crudrouter

I can't show any pics cuz of uk NDA and all that. I can tell you the gist of it though. FastAPI is already pretty good at the abstraction part. Our middleware had a lot of rerouting and it was basically just a bunch of redundant functions. I just used this cool ass package https://fastapi-crudrouter.awtkns.com/ and used the partial function from functools to generate endpoints for every scenario/db tables.

Thanks, that's a really helpful example.

Where I think this could be taken to the next level of reusability is in modularising the front-end into API-specific components. For example, the login behaviour could depend on FastAPI-Users, with a sibling frontend library containing components that implement the same login flow. Adding user behaviour is then a matter of using the same third-party library on the front and back end.

This approach could be extended to other components such as an admin panel (perhaps using https://github.com/awtkns/fastapi-crudrouter), or a blogging component.