Airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows (by apache)

Airflow Alternatives

Similar projects and alternatives to Airflow

  • GitHub repo black

    The uncompromising Python code formatter

  • GitHub repo pre-commit

    A framework for managing and maintaining multi-language pre-commit hooks.

  • GitHub repo youtube-dl

    Command-line program to download videos from YouTube.com and other video sites

  • GitHub repo Moby

    Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

  • GitHub repo scikit-learn

    scikit-learn: machine learning in Python

  • GitHub repo requests

    A simple, yet elegant HTTP library.

  • GitHub repo rich

    Rich is a Python library for rich text and beautiful formatting in the terminal.

  • GitHub repo Docker Compose

    Define and run multi-container applications with Docker

  • GitHub repo MongoDB

    The MongoDB Database

  • GitHub repo n8n

    Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services.

  • GitHub repo Poetry

    Python dependency management and packaging made easy.

  • GitHub repo FizzBuzz Enterprise Edition

    FizzBuzz Enterprise Edition is a no-nonsense implementation of FizzBuzz made by serious businessmen for serious business purposes.

  • GitHub repo PostgreSQL

    Mirror of the official PostgreSQL GIT repository. Note that this is just a *mirror* - we don't work with pull requests on github. To contribute, please see https://wiki.postgresql.org/wiki/Submitting_a_Patch

  • GitHub repo falcon

    The no-nonsense REST API and microservices framework for Python developers, with a focus on reliability, correctness, and performance at scale.

  • GitHub repo StackStorm

    StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html. Questions? https://forum.stackstorm.com/.

  • GitHub repo SQLAlchemy

    The Database Toolkit for Python

  • GitHub repo Apache Camel

    Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.

  • GitHub repo Maven Wrapper

    The easiest way to integrate Maven into your project!

  • GitHub repo 0bin

    Client side encrypted pastebin

  • GitHub repo code

    Example application code for the python architecture book

NOTE: The number of mentions on this list indicates mentions on common posts. Hence, a higher number means a better Airflow alternative or higher similarity.

Posts

Posts where Airflow has been mentioned. We have used some of these posts to build our list of alternatives and similar projects - the last one was on 2021-07-05.
  • Install Airflow 2 on a Raspberry Pi (using Python 3.x)
    dev.to | 2021-07-22
    Go to Airflow's github repo and download the airflow-webserver.service and the airflow-scheduler.service
  • That one question again: an alternative to Jenkins?
    reddit.com/r/devops | 2021-07-05
    Circle CI has positioned themselves as a modern Jenkins replacement, but I should note I haven't seen them used over a long period of time which is the real measure of quality. Honestly most pipelines seem to be better suited to workflow frameworks like Apache Airflow than these CI/CD specific tools, they may be a little harder to learn up front but they seem to be more resilient to the type of customization that usually gets you in trouble with Jenkins down the line.
  • Occasional "No such file" errors even though file exists when trying to wget files from FTP?
    reddit.com/r/sysadmin | 2021-06-22
    Occasionally, we get "No such file" errors, even though the file does exist on the FTP server (when inspecting via FTP GUI), when trying to import. Note, this does not happen all the time nor for the same files each time. For context, we have an airflow (https://airflow.apache.org/) scheduled process (running on "local" executor mode) that imports multiple TSV files from an FTP server in a multithreaded manner in pools of 3 at a time.
  • New to data orchestration? Start here.
    dev.to | 2021-06-02
    First-generation data orchestration tools like Airflow are primarily focused on improving usability for data scientists with the introduction of Python support (vs previous tools that required queries to be written in JSON and YAML). This improved UI made it easier for data teams to manage their pipeline flows without getting as caught up in the process.
  • BigQueryInsertJobOperator missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential.
    u/The_Fog_Bandit OK, now I'm wondering if the issue is actual something else. Because in my logs, it says "Required parameter is missing" . I assumed its the oauth2 because when I click on the link in my logs it says required oauth2 missing, but perhaps that's not the error message that pertains to my DAG issue, it just populates because I haven't entered in credentials to access that link?! but I followed this: https://github.com/apache/airflow/blob/25caeda58b50eae6ef425a52e794504bc63855d1/airflow/providers/google/cloud/operators/bigquery.py#L2137
  • Docker Swarm cron job manager
    reddit.com/r/docker | 2021-05-28
    I came across https://airflow.apache.org/ as I think is a bit overkill for my needs.
  • Wanted: Scheduled tasks, initiated by Agent, then managed separately
    reddit.com/r/SQL | 2021-05-17
  • What are the differences between Astronomer's airflow chart and the community one? Which one is better?
    There's also Airflow's official Helm chart - https://github.com/apache/airflow/tree/master/chart.
  • airflow.exceptions.AirflowException: Argument ['configuration'] is required
    The error message, which you've correctly read, points at line 64 as the place that's an issue. The error also says Argument ['configuration'] is required, so you haven't provided an argument when calling something. Given that the error notes BigQueryInsertJobOperator, you can check the source code to see what's expected.
    u/The_Fog_Bandit oh shoot! Im sorry you're right! we were previously using the BigQueryOperator class, and I incorrectly change it to BigQueryInsertJobOperator but it should have been BigQueryExecuteQueryOperator https://github.com/apache/airflow/blob/master/airflow/contrib/operators/bigquery_operator.py I honestly don't know why I thought it was BigQueryInsertJobOperator thank you so much for taking a look at my question and answering it, I would have used the wrong operator!
  • Airflow 404 = lots of circles
    I checked the config file example for airflow here and it matches: https://github.com/apache/airflow/blob/25caeda58b50eae6ef425a52e794504bc63855d1/airflow/config_templates/default_airflow.cfg#L381
  • 'airflow.providers.google.cloud.hooks.gcs' is not a package
    That file you've pointed to isn't telling you to do what you've done: https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/gcs_hook.py#L21
  • Im making a FME open source clone
    reddit.com/r/gis | 2021-05-05
    I have heard good things about Apache Airflow, so maybe developing GIS plugins/extensions to that might have a broader reach: http://airflow.apache.org/
  • Do you know any Python projects on Github that are examples of best practices and good architecture?
    This repo doesn't have a main function, so it's not supposed to run by itself, but instead is a pipeline for Apache Airflow. But that isn't what makes this code good.

Stats

Basic Airflow repo stats
42
22,472
10.0
4 days ago

apache/airflow is an open source project licensed under Apache License 2.0 which is an OSI approved license.