mlrun
Prefect
Our great sponsors
mlrun | Prefect | |
---|---|---|
3 | 19 | |
1,240 | 14,330 | |
4.8% | 3.0% | |
9.9 | 9.9 | |
1 day ago | 5 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mlrun
-
I reviewed 50+ open-source MLOps tools. Here’s the result
You should also add MLRun: https://github.com/mlrun/mlrun
Prefect
- self hosted Alternative to easycron.com?
-
Example typescript project repos?
If I was answering this question but for python, I'd recommend something like prefect, boto3, or tortoise-orm -- not extremely complex and with a pretty comprehensible featureset.
-
I have developed a simple Task Orchestrator
However, if you are looking for something like this, but much more mature and something of a bloat to be frank, there's Prefect. Honestly, woflo borrows a lot from Prefect conceptually.
-
Dabbling with Dagster vs. Airflow
Disclaimer: I work for Prefect.
It looks like we added cron and other schedule types to the deployment CLI just under a month ago[1].
Over the last couple of releases, we've also made it easier to pull deployments from GitHub or bake your flow code into Docker images instead of needing S3-like storage.
As with any product, there's always more to do, so I appreciate you sharing your thoughts. More than anywhere else I've worked, community feedback is a huge driver of product enhancements and feature development. Feel free to join our Slack community[2] if you'd like to share more feedback or ask questions.
[1] https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES...
- Ask HN: Codebases with great, easy to read code?
-
Prefect CLI Action
GitHub Action for running Prefect commands using the Prefect CLI.
-
My first Hacktoberfest !!✨
View on GitHub
-
6502 - Some Thoughts on Open Source Projects
Here, I am going to pick two open source packages to talk and compare with. Pandas VS. Prefect.
-
Hi, it's me, your mom, trying to get back to work after not doing tech for almost 20 years while I raised you and your siblings
From my point of view the experience will be much better and there are many things that are done with it right now, including most automation tools for deploying and pipelines (prefect.io, Airflow, and others), data processing, machine learning and so on.
-
Hi, how can I do pipeline automation?
If you are just starting out or new to doing automation, I would look at just python scripts executed with CRON if on Linux/Mac or Windows Task Scheduler if on Windows. But you'll need bash (Linux/Mac) knowledge or DOS/batch knowledge (Windows). Then graduate to using frameworks. Since you didnt specify what types of jobs you want to automate, for general purpose needs, I would look at a class of frameworks called task orchestration frameworks or workflow management libraries. I would highly recommend dagster as it comes with a native scheduler so you would be free from having to use CRON or Windows Task Scheduler. Other options include prefect, but if you want its other features like its scheduler and web GUI, you'll have to mess with docker. That's what's nice about dagster, it all works out of the box without need for non-Python dependencies.
What are some alternatives?
dagster - An orchestration platform for the development, production, and observation of data assets.
APScheduler - Task scheduling library for Python
airbyte - The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
schedule - Python job scheduling for humans.
doit - task management & automation tool
django-schedule - A calendaring app for Django. It is now stable, Please feel free to use it now. Active development has been taken over by bartekgorny.
fastapi-dramatiq-data-ingestion - Sample project showing reliable data ingestion application using FastAPI and dramatiq
Joblib - Computing with Python functions.
spark-fast-tests - Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
TaskFlow - A library to complete workflows/tasks in HA manner. Mirror of code maintained at opendev.org.
meltano - Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Spiff - A powerful workflow engine implemented in pure Python