cargo-crates
Prefect
cargo-crates | Prefect | |
---|---|---|
3 | 19 | |
1 | 14,829 | |
- | 3.0% | |
3.1 | 10.0 | |
about 1 month ago | 3 days ago | |
Python | Python | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cargo-crates
-
Docker - Magic or Hype?
I've used this benefit in one of my personal side projects (cargo-crates) to have ready-made containers for data extraction purposes. I'm always picking up projects and putting them back down, or shifting which versions of different libraries I have on my laptop, so picking up an old project with specific library dependencies can be really annoying.
-
Your default tool for ETL
I went a little crazy and built my own set of data extractors that I can deploy with CDK to ECS.
-
Why is it so hard to think of a DE side project idea ?
- Extract data from system. I wear an Oura ring for sleep tracking. I wanted to do my own analysis of the data, so I built a system that could easily allow me to extract the data into S3 so I could query it. https://github.com/dacort/cargo-crates Will anybody find that useful? Maybe...but it's been a heck of a lot of fun and really pushed my Docker skills.
Prefect
- Prefect: A workflow orchestration tool for data pipelines
- self hosted Alternative to easycron.com?
-
Example typescript project repos?
If I was answering this question but for python, I'd recommend something like prefect, boto3, or tortoise-orm -- not extremely complex and with a pretty comprehensible featureset.
-
I have developed a simple Task Orchestrator
However, if you are looking for something like this, but much more mature and something of a bloat to be frank, there's Prefect. Honestly, woflo borrows a lot from Prefect conceptually.
-
Dabbling with Dagster vs. Airflow
Disclaimer: I work for Prefect.
It looks like we added cron and other schedule types to the deployment CLI just under a month ago[1].
Over the last couple of releases, we've also made it easier to pull deployments from GitHub or bake your flow code into Docker images instead of needing S3-like storage.
As with any product, there's always more to do, so I appreciate you sharing your thoughts. More than anywhere else I've worked, community feedback is a huge driver of product enhancements and feature development. Feel free to join our Slack community[2] if you'd like to share more feedback or ask questions.
[1] https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES...
- Prefect - The easiest way to automate your data
- Ask HN: Codebases with great, easy to read code?
-
Prefect CLI Action
GitHub Action for running Prefect commands using the Prefect CLI.
- Perfect – Data workflow automation with Python
What are some alternatives?
dbt-spark - dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
dagster - An orchestration platform for the development, production, and observation of data assets.
airflow-docker - This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
APScheduler - Task scheduling library for Python
airbyte - The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Apache Superset - Apache Superset is a Data Visualization and Data Exploration Platform [Moved to: https://github.com/apache/superset]
schedule - Python job scheduling for humans.
doit - task management & automation tool
django-schedule - A calendaring app for Django. It is now stable, Please feel free to use it now. Active development has been taken over by bartekgorny.
fastapi-dramatiq-data-ingestion - Sample project showing reliable data ingestion application using FastAPI and dramatiq
Joblib - Computing with Python functions.