airflowctl
astronomer-cosmos
| airflowctl | astronomer-cosmos | |
|---|---|---|
| 1 | 8 | |
| 169 | 461 | |
| - | 6.5% | |
| 8.0 | 9.4 | |
| 6 days ago | 4 days ago | |
| Python | Python | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
airflowctl
-
A look at airflowctl, a tool to help developers manage Apache Airflow projects
NOTE! I found a small issue in that when you run in background mode, it creates a file (.airflowctl/.background_process_ids) which contains the parent PID. The PID was always off, so I needed to manually edit this. I have created an issue here so if this happens to you, follow that.
astronomer-cosmos
- Run dbt projects as Apache Airflow DAGs and Task Groups with a few lines of code
-
Running dbt core on airflow
You could also try cosmos! https://astronomer.github.io/astronomer-cosmos/
-
What's the best way to learn dbt
And rather than a bash operator to trigger your dbt job you can use cosmos to dynamically generate your dbt model as an Airflow job https://astronomer.github.io/astronomer-cosmos/
-
PSA: we learned the hard way DBT Cloud support doesn’t work weekends…
Especially now Cosmos is helping to integrate your dbt projects like never before https://github.com/astronomer/astronomer-cosmos
-
Best Orchestration Tool to run dbt projects?
Well, I would take a look at Astronomer Cosmos which is a framework to dynamically Airflow DAGs from dbt with a single operator.No need to manage connections, use true BashOperator etc. It’s dead simple and open source https://astronomer.github.io/astronomer-cosmos/ I’m going to make a video about it very soon.
-
ELT pipeline using airflow
- Astronomer Cosmos*: Makes converting dbt-core projects into Airflow DAGs drop dead simple
-
dbt-cloud along side orchestrator (like AirFlow) in the same company
Check this out: Astronomer Cosmos
-
dbt Cloud Alternatives?
With v0.3.0 that we shipped today, we added select/exclude parameters on DbtDag and DbtTaskGroup so that you can filter down to a specific set of models based on dbt tags. In the next couple of weeks, we will add functionality to use select/exclude to filter on Configs, Paths, and Sources. For more info, see the GitHub here (the /examples directory has DAG code that demonstrates DbtDag and DbtTaskGroup
What are some alternatives?
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
astro-sdk - Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
incubator-airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows [Moved to: https://github.com/apache/airflow]
airflow-maintenance-dags - A series of DAGs/Workflows to help maintain the operation of Airflow
elyra - Elyra extends JupyterLab with an AI centric approach.
astro - Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow. [Moved to: https://github.com/astronomer/astro-sdk]
soda-sql - Data profiling, testing, and monitoring for SQL accessible data.
data-aware-orchestration - Data-aware orchestration with dagster, dbt, and airbyte