airflow-maintenance-dags
astronomer-cosmos
| airflow-maintenance-dags | astronomer-cosmos | |
|---|---|---|
| 1 | 8 | |
| 1,599 | 457 | |
| 0.9% | 4.0% | |
| 0.0 | 9.4 | |
| 3 months ago | 1 day ago | |
| Python | Python | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
airflow-maintenance-dags
-
Airflow meta-db cleaning
Use these maintenance DAGs from the project https://github.com/teamclairvoyant/airflow-maintenance-dags
astronomer-cosmos
- Run dbt projects as Apache Airflow DAGs and Task Groups with a few lines of code
-
Running dbt core on airflow
You could also try cosmos! https://astronomer.github.io/astronomer-cosmos/
-
What's the best way to learn dbt
And rather than a bash operator to trigger your dbt job you can use cosmos to dynamically generate your dbt model as an Airflow job https://astronomer.github.io/astronomer-cosmos/
-
PSA: we learned the hard way DBT Cloud support doesn’t work weekends…
Especially now Cosmos is helping to integrate your dbt projects like never before https://github.com/astronomer/astronomer-cosmos
-
Best Orchestration Tool to run dbt projects?
Well, I would take a look at Astronomer Cosmos which is a framework to dynamically Airflow DAGs from dbt with a single operator.No need to manage connections, use true BashOperator etc. It’s dead simple and open source https://astronomer.github.io/astronomer-cosmos/ I’m going to make a video about it very soon.
-
ELT pipeline using airflow
- Astronomer Cosmos*: Makes converting dbt-core projects into Airflow DAGs drop dead simple
-
dbt-cloud along side orchestrator (like AirFlow) in the same company
Check this out: Astronomer Cosmos
-
dbt Cloud Alternatives?
With v0.3.0 that we shipped today, we added select/exclude parameters on DbtDag and DbtTaskGroup so that you can filter down to a specific set of models based on dbt tags. In the next couple of weeks, we will add functionality to use select/exclude to filter on Configs, Paths, and Sources. For more info, see the GitHub here (the /examples directory has DAG code that demonstrates DbtDag and DbtTaskGroup
What are some alternatives?
astro - Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow. [Moved to: https://github.com/astronomer/astro-sdk]
astro-sdk - Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
airflowctl - A CLI tool to streamline getting started with Apache Airflow™ and managing multiple Airflow projects
ask-astro - An end-to-end LLM reference implementation providing a Q&A interface for Airflow and Astronomer
incubator-airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows [Moved to: https://github.com/apache/airflow]
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
airflow-chart - A Helm chart to install Apache Airflow on Kubernetes
soda-sql - Data profiling, testing, and monitoring for SQL accessible data.
data-aware-orchestration - Data-aware orchestration with dagster, dbt, and airbyte