rocketry
Mage
rocketry | Mage | |
---|---|---|
32 | 77 | |
3,178 | 7,050 | |
- | 3.5% | |
0.6 | 9.9 | |
6 months ago | 2 days ago | |
Python | Python | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rocketry
-
Is SQL a prerequisite to learning Python ?
Intuitive scheduling framework: https://rocketry.readthedocs.io/
-
Easiest way to run a scheduled Python script?
One easy option would be to use Rocketry, it's pure Python:
-
Scheduling in Data Engineering
It has bunch of features: logical scheduling syntax, parallelism/concurrency, dynamic parametization, log to database etc. You can read from the docs: https://rocketry.readthedocs.io/
-
Run external programs intuitively in Python
It's the creator of Rocketry, Red Mail and Red Box again. This week I thought to make it easier to integrate command-line programs to your Python applications.
-
Should i use airflow
If you don't need fancy UIs (or you can create those yourself), I have heard some people are replacing Airflow with Rocketry. Rocketry is very easy to set up and use and its scheduling mechanics are pretty advanced.
-
Email Utility
If you need a scheduler (that also supports custom conditions like "run when there is X amount of emails"), I have another library for that: Rocketry
-
Trying to send Email
Smtplib is a pain to work with (which is the reason I wrote Red Mail). For the scheduling, I also happen to have a library that is often useful for this sorts of problems: https://rocketry.readthedocs.io
-
WHAT ENGINE IS THE BEST FOR PROTOTYPING?(Python)
I have been thinking of creating an algo using Rocketry at some point. It's very easy to create custom conditions and combine these with logical statement. For example, you could set your algo (a function) running when volatility is x amount and you have y amount balance.
-
I have developed a simple Task Orchestrator
I have been tackling the same problem and it has recently gained popularilty. In case you want to take a look at Rocketry: https://github.com/Miksus/rocketry
- A statement-based scheduling framework for Python
Mage
- FLaNK AI-April 22, 2024
-
A mage on the Hero’s Journey: a fantasy epic on how a startup rose from the ashes
In the coming years, Mage will create a cooperative experience so that developers can build data pipelines with their team and level up together. After that journey, Mage will go on an epic quest to create the 1st open world community experience in the data universe.
-
Data sources episode 2: AWS S3 to Postgres Data Sync using Singer
Link to original blog: https://www.mage.ai/blog/data-sources-ep-2-aws-s3-to-postgres-data-sync-using-singer
-
What are some open-source ML pipeline managers that are easy to use?
I would recommend the following: - https://www.mage.ai/ - https://dagster.io/ - https://www.prefect.io/ - https://metaflow.org/ - https://zenml.io/home
-
Mage Battlegrounds: Craft insights from real-time customer behavior analysis
You're invited to participate in the very first Mage Battlegrounds: Craft insights from real-time customer behavior analysis, a 24-hour virtual hackathon hosted by Shashank Mishra! This data engineering competition will take place on Saturday, April 15, 2023 beginning at 11am (PST). This will be a global event open to all participants who register.
-
Looking for an open-source project
Try this feature: https://github.com/mage-ai/mage-ai/issues/1166
-
Daskqueue: Dask-based distributed task queue
Seeing if we can use it in https://github.com/mage-ai/mage-ai
-
Data Pipeline on a Shoestring
That being said there’s a solid family of services just breaking ground that make the local pipeline deployment easier (check out https://www.mage.ai, which does have a clear path to cloud deployment of locally developed pipes, it just isn’t well documented yet, and also https://www.neuronsphere.io - which doesn’t have a public solution YET (they’re internally testing an alpha) but they built a cloud deployable solution for their paying customers and working to release one for freemium use)
-
Trending ML repos of the week 📈
7️⃣ mage-ai/mage-ai
-
Delta without using Spark
Yes, check out how Mage does it: https://github.com/mage-ai/mage-ai/tree/master/mage_integrations/mage_integrations/destinations/delta_lake_s3
What are some alternatives?
Python Fire - Python Fire is a library for automatically generating command line interfaces (CLIs) from absolutely any Python object.
dagster - An orchestration platform for the development, production, and observation of data assets.
rocketry-with-fastapi - Example of how to create a scheduler with an API
vscode-dvc - Machine learning experiment tracking and data versioning with DVC extension for VS Code
django-formset - The missing widgets and form manipulation library for Django
sqlmesh - Efficient data transformation and modeling framework that is backwards compatible with dbt.
scheduled_thread_pool_executor - Scheduled Thread Pool Executor implementation in python
mito - The mitosheet package, trymito.io, and other public Mito code.
protobuf_to_pydantic - Generate a pydantic.BaseModel with parameter verification function from the Python Message object(by the Protobuf file).
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
red-mail - Advanced email sending for Python
Data-Science-Roadmap - Data Science Roadmap from A to Z