Dataplane
data-engineering-wiki
| Dataplane | data-engineering-wiki | |
|---|---|---|
| 1 | 15 | |
| 184 | 1,031 | |
| 1.1% | 2.9% | |
| 8.3 | 7.5 | |
| 4 months ago | about 1 month ago | |
| Go | CSS | |
| Business Source License 1.1 | Creative Commons Zero v1.0 Universal |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Dataplane
-
Airflow VS dataplane - a user suggested alternative
2 projects | 3 May 2022
Dataplane is an Airflow inspired data platform to automate, schedule and design data pipelines and workflows written in Golang.
data-engineering-wiki
- Data Engineering Glossary
-
ETL practice
My suggestions: 1. Browse https://dataengineering.wiki/ and overall go over r/dataengineering 2. In mid-sized companies, the trend is to outsource Extract and Load to providers like Fivetran or Airbyte (open-source). Then Transform it with dbt in a data warehouse with SQL. 3. In big companies, you won't touch much ETL design. Just need to be proficient in Python / Spark / SQL... 4. Make sure you know what a star schema, fact tables, and dimension tables are.
- Anything else to read
-
Looking for blogs for backend development
Hi everyone! As mentioned in title I recently came across great blogs for data engineering: startdataengineering.com and dataengineering.wiki
-
DE- How to get my foot in the door?
The data engineering subreddit maintains a wiki of advice, resources, and recommendations at https://dataengineering.wiki/. Your question is answered in their FAQ here
- Getting into Data Engineering and more!
-
Are there avenues into sports science as a software engineer or web dev?
Data engineering
-
Switching to something more technical
r/dataengineering has a wiki at https://dataengineering.wiki and also a Discord server which is pretty active.
-
Data Engineering Concepts: Definitions, Backlinks, and Graph View
Almost the same as the wiki https://dataengineering.wiki/
-
dataengineering.wiki Bug
Hi, would you mind opening an issue on GitHub? We can help you debug the issue there.
What are some alternatives?
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
glossary - Data Glossary 🧠: An interactive digital garden for deeper data exploration. Learn through a graph and backlinks, enabling layered knowledge discovery.
dagu - Yet another cron alternative with a Web UI, but with much more capabilities. It aims to solve greater problems.
versatile-data-kit - One framework to develop, deploy and operate data workflows with Python and SQL.
transfer - Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
quartz - 🌱 a fast, batteries-included static-site generator that transforms Markdown content into fully functional websites
JDR - Job Dependency Runner
sayn - Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
dagster - An orchestration platform for the development, production, and observation of data assets.
Mage - 🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
starthinker - Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."
Hugo - The world’s fastest framework for building websites.