Our great sponsors
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I am currently using dagster as a workflow orchestration/ETL tool and it is such a fantastic framework. It allows DAs or DSs to be more full-stack. I think using something like dagster is the future. It is cross-platform and acutally works well on Windows, so even a data analyst or data scientist in a small fry corporate Windows environment can become more full-stack with dagster. So I can see it having more widespread use by more people in various industries, not just tech-focused industries where they are using exclusive non-Windows stack. When they are teamed up with experienced software engineers, their projects can scale if needed.
I have created a trivial, contrived scikit-learn example using dagster so that people have an idea of how it can be used.