zpy
Airflow
zpy | Airflow | |
---|---|---|
35 | 169 | |
67 | 34,570 | |
- | 1.4% | |
9.1 | 10.0 | |
2 days ago | 4 days ago | |
Shell | Python | |
Do What The F*ck You Want To Public License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zpy
-
This Week In Python
zpy – Zsh helpers for Python venvs, with uv or pip-tools
-
Canonical blocked installing, or uninstalling pip packages on Ubuntu 23.04, what it can be done to solve these issues?
If your interactive shell is zsh, you could give my project zpy a try, particularly the function pipz that it provides, which is a lightweight pipx clone with great completions and good speed.
-
As if there weren't enough packaging tools already: mitsuhiko/rye: an experimental alternative to poetry/pip/pipenv/venv/virtualenv/pdm/hatch/…
I can immediately see some things rye is doing differently, like keeping the venvs themselves free of pip and pip-tools. I wonder in your explorations if you've tried rtx for managing python installations, or my own zpy wrapper of pip-tools+venv (which can also replace pipx).
-
How do I build up my package's extra dependencies from groups of dependencies in a pyproject.toml?
My patterns in this regard aren't exactly mainstream, as I use flit+pip-tools+zpy (the latter being my own Zsh interface for Python dependency and environment operations), but FWIW here's how I go about nested requirements.
-
What is your workflow for managing virtual environments for personal projects?
For managing venvs and dependencies and apps, I use my own frontend to pip-tools + venv, zpy. And for running tasks which require an activated venv, I use nox.
-
One Does Not Simply 'pip install'
If anyone's interested in a pipx clone with excellent tab completion, I would appreciate any feedback on pipz, a function of my zsh plugin for python environment and dependency management: zpy
https://github.com/andydecleyre/zpy
-
pipenv or virtualenv ?
For concise and practical interactive usage of those tools, with excellent tab completion, I made the Zsh frontend zpy.
-
How to know what a package depend on when pip is installing it?
I also use my own Zsh wrapper functions with it, so for example: https://i.imgur.com/YX8bWy8.png
-
I moved away from Poetry for Python
I'm a big fan of (and small contributor to) pip-tools, but both poetry and pipenv offer management of more stuff, which understandably appeals to folks seeking a simple comprehensible workflow.
Pip-tools is also a bit lower level, offering flexibility and compatibility which I relish, but also requiring more attention from the user to set things up as they wish.
If you or anyone else enjoying pip-tools is a Zsh user and interested in trying out my higher level functions to ease interactive use of pip-tools, venvs, and also isolated app installs (like pipx), I would love some feedback on zpy: https://github.com/AndydeCleyre/zpy
I'm very happy to answer any questions about it right here or as GitHub issues.
-
Any recent updates in dependency management?
This is FAR from some big mainstream thing, but I use (and am happy to answer any questions about) my own Zsh frontend to venv+pip-tools+pip, zpy.
Airflow
-
Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions
Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.
-
Navigating Week Two: Insights and Experiences from My Tublian Internship Journey
In week Two, I contributed to the Apache Airflow repository.
-
Airflow VS quix-streams - a user suggested alternative
2 projects | 7 Dec 2023
-
Best ETL Tools And Why To Choose
Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform features a web-based user interface and a command-line interface for managing and triggering workflows.
-
Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow
Airflow is the most widely used and well-known tool for orchestrating data workflows. It allows for efficient pipeline construction, scheduling, and monitoring.
-
Share Your favorite python related software!
AIRFLOW This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic!
-
Ask HN: What is the correct way to deal with pipelines?
I agree there are many options in this space. Two others to consider:
- https://airflow.apache.org/
- https://github.com/spotify/luigi
There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory…
- "Você veio protestar para ter acesso ao código fonte da urnas. O que é o código fonte?" "Não sei" 🤡
- Cómo construir tu propia data platform. From zero to hero.
-
Is it impossible to contribute to open source as a data engineer?
You can try and contribute some new connectors/operators for workflow managers like Airflow or Airbyte
What are some alternatives?
hatch - Modern, extensible Python project management
Kedro - Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
agkozak-zsh-prompt - A fast, asynchronous Zsh prompt with color ASCII indicators of Git, exit, SSH, virtual environment, and vi mode status. Framework-agnostic and customizable.
dagster - An orchestration platform for the development, production, and observation of data assets.
wheezy.template - A lightweight template library.
n8n - Free and source-available fair-code licensed workflow automation tool. Easily automate tasks across different services.
taskipy - the complementary task runner for python
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
zplug - :hibiscus: A next-generation plugin manager for zsh
Apache Spark - Apache Spark - A unified analytics engine for large-scale data processing
tox-pin-deps - Run tox environments with strictly pinned dependencies (and no project or code changes).
Dask - Parallel computing with task scheduling