docker-airflow
catalog
Our great sponsors
docker-airflow | catalog | |
---|---|---|
10 | 22 | |
3,703 | 1,432 | |
- | 0.6% | |
0.0 | 8.2 | |
about 1 year ago | about 1 month ago | |
Shell | TypeScript | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
docker-airflow
-
Kubernetes deployment read-only filesystem error
I am facing an error while deploying Airflow on Kubernetes (precisely this version of Airflow https://github.com/puckel/docker-airflow/blob/1.8.1/Dockerfile) regarding writing permissions onto the filesystem.
-
How to use virtual environment in airflow DAGS?
I used https://github.com/puckel/docker-airflow to setup the airflow and I moved my python scripts inside the dags directory but now they won't execute because I can't access the installed libraries in the virtual environment. How can i find a workaround for this?
-
Amount of effort to stand up, integrate and manage a small airflow implementation
Used a custom version of Puckel Airflow Docker image (Spent a lot of time customising to our needs, but default Airflow container should still work)
-
The Unbundling of Airflow
I understand it is subjective. But I use a forked version of https://github.com/puckel/docker-airflow on our managed K8s cluster and it points to a cloud managed Postgres. It has worked pretty well for over 3 years with no-one actually managing it from an infra POV. YMMV. This is driving a product whose ARR is well in the 100s of Millions.
If you have simple needs that are more or less set, I agree Airflow is overkill and a simple Jenkins instance is all you need.
-
Airflow v1 to v2 - Recommendations / RoX
So were running Airflow v1 (based on this docker compose) with a sequential executor running on an on prem OpenShift v3 setup. We have a new / free resource coming and have planned to use them to reinitiate a complete new version utilizing OpenShift v4 (also on prem but not managed by us) and upgrade in parallel to Airflow v2. The question is if anyone has any strong recommendations on a good docker compose file they would look at and any views on celery / kubernets workers. We're not a huge team but have a bit of experience up our sleeves now so was more after some guidance or thoughts if others have gone down similar paths. Thanks!
-
Can someone help me understand the difference between the the docker-compose files?
version: '3' services: postgres: image: postgres:9.6 environment: - POSTGRES_USER=airflow - POSTGRES_PASSWORD=airflow - POSTGRES_DB=airflow ports: - "5432:5432" webserver: image: puckel/docker-airflow:1.10.1 build: context: https://github.com/puckel/docker-airflow.git#1.10.1 dockerfile: Dockerfile args: AIRFLOW_DEPS: gcp_api,s3 PYTHON_DEPS: sqlalchemy==1.2.0 restart: always depends_on: - postgres environment: - LOAD_EX=n - EXECUTOR=Local - FERNET_KEY=jsDPRErfv8Z_eVTnGfF8ywd19j4pyqE3NpdUBA_oRTo= volumes: - ./examples/intro-example/dags:/usr/local/airflow/dags # Uncomment to include custom plugins # - ./plugins:/usr/local/airflow/plugins ports: - "8080:8080" command: webserver healthcheck: test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"] interval: 30s timeout: 30s retries: 3
-
How should I get started with CI/CD ? (new to data engineering)
As for learning, learn how to build and use docker containers. For airflow, take a look a https://github.com/puckel/docker-airflow and see how to add you pipelines to that container. Then learn how to do CI/CD for docker containers (tons of tutorials). Then learn to deploy containers, you can use aws ecs.
-
Interview - take home project on data ingestion, warehouse design, basic analytics and conceptual using python and sql.
Usually googling the software you want + docker will get you what you need. For that particular project, I used https://github.com/puckel/docker-airflow to help set up a local airflow instance.
-
ETL com Apache Airflow, Web Scraping, AWS S3, Apache Spark e Redshift | Parte 1
A imagem do docker utilizada foi a puckel/docker-airflow onde acrescentei o BeautifulSoup como dependência para criação da imagem em minha máquina.
-
How we evolved our data engineering workflow day by day
We used to schedule and monitor workflows tool airflow as our ELT processor and have to extract data from SQL and No-SQL databases to load them into the warehouse. Our airflow deployment was done through docker, for more details checkout puckel/airflow. Currently, we are adopting our image to the official docker images.
catalog
- Mirrorful – free and open-source React component library
-
Lot of discussions about setlists this tour, but at 63 songs across 10 concerts so far, I think... baby, we'll be fine.
If you're at all into open-source, the code for this script is here on Github. Feel free to give it a star or make contributions. I want to improve it, but my full-time job is managing another open-source project.
-
Css 500 different hex colors
You should checkout this tool Mirrorful that solves this exact problem! Disclaimer: I’m one of the cofounders - feel free to DM me!
- Mirrorful - a simple, open-source design system framework. Install Mirrorful to generate colors and other design tokens for your project. Then, import these tokens directly into your app
-
Show HN: Mirrorful – A developer-first way to implement designs faster
The files that we currently export have CSS variables that once imported, can be used anywhere in your app! With this set up, instead of needing to reference arbitrary hex colors throughout your CSS, you now have a code-based source of truth for your colors and design tokens.
For existing apps, there is definitely a bit of a migration process, appreciate that callout! The good news is that its a migration that can happen over time, as using Mirrorful isn't dependent on your whole codebase being consistent on day 1. We're also looking to leverage AI to do auto-migrations!
We currently don't have a direct integration with Bootstrap, but we definitely work with Bootstrap! I just created a Github Issue for us to create an example to show how you can use Mirrorful with Bootstrap: https://github.com/Mirrorful/mirrorful/issues/260
-
Comments on this TikTok talking about how web dev is not "software engineering"?!
Here's the link to the TikTok to see the comments: https://www.tiktok.com/t/ZTRvbM76p/ He talks about value of design systems tooling, was talking about this: https://github.com/Mirrorful/mirrorful I thought it was interesting how one of the comments calls out web dev
-
I built a platform that generates CSS variables for your theme!
This project is also open-source, check out the Github here! It's super early so would love to hear any thoughts or feedback to help shape what to build next!
We’re also open-source, check out our Github here!
What are some alternatives?
orchest - Build data pipelines, the easy way 🛠️
ontime - Free, open-source time keeping for live events
ploomber - The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
design-ops-hackpack - Template repo for bootstrapping a desOps practice using github.
wordpress-docker-compose - Easy Wordpress development with Docker and Docker Compose
deno - A modern runtime for JavaScript and TypeScript.
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
theme-ui - Build consistent, themeable React apps based on constraint-based design principles
beginner_de_project - Beginner data engineering project - batch edition
bun - Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one
catalog - Catalog of shared Tasks and Pipelines.
librephotos-frontend - A self-hosted open source photo management service. This is the repository of the frontend.