|9 months ago||4 days ago|
|MIT License||GNU General Public License v3.0 or later|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Well designed scala/spark project
4 projects | reddit.com/r/scala | 15 Oct 2022
Unit & integration testing in Databricks
3 projects | reddit.com/r/dataengineering | 30 Apr 2022
If the majority of your stuff is not UDF-based there is an OS solution to run assertion tests against full data frames called spark-fast-tests. The idea here is similar in that you have a it notebook that calls your actual notebook against a staged input reads the output and compares it to a prefabed expected output. This does take a bit of setup and trial and error but it’s the closest I’ve been able to get to proper automated regression testing in databricks
Show dataengineering: beavis, a library for unit testing Pandas/Dask code
3 projects | reddit.com/r/dataengineering | 9 Aug 2021
I am the author of spark-fast-tests and chispa, libraries for unit testing Scala Spark / PySpark code.
Ask HN: What are some tools / libraries you built yourself?
264 projects | news.ycombinator.com | 16 May 2021
I built daria (https://github.com/MrPowers/spark-daria) to make it easier to write Spark and spark-fast-tests (https://github.com/MrPowers/spark-fast-tests) to provide a good testing workflow.
Built bebe (https://github.com/MrPowers/bebe) to expose the Spark Catalyst expressions that aren't exposed to the Scala / Python APIs.
Also build spark-sbt.g8 to create a Spark project with a single command: https://github.com/MrPowers/spark-sbt.g8
Open source contributions for a Data Engineer?
17 projects | reddit.com/r/dataengineering | 16 Apr 2021
I've built popular PySpark (quinn, chispa) and Scala Spark (spark-daria, spark-fast-tests) libraries.
What are your thoughts on projects using the Elastic License?
2 projects | reddit.com/r/opensource | 26 Jan 2023
Doing a quick GitHub search reveals quite a few projects using the ELv2 license, including Airbyte and InvoiceNinja. Elastic (the company) aside, what are your thoughts on the Elastic License v2? Does your employer allow projects with an ELv2 license? Do you consider it open source? I understand that it's not OSI approved, but wondering where people stand when it comes to commercial open source software.
Airbyte Source Connectors performance bottelneck
2 projects | reddit.com/r/dataengineering | 15 Jan 2023
I have been using airbyte sources, S3 mainly, it is so slow, I'm getting 1k-3k records per second, on a high end machine 4 Cpus and 16GB Ram. I checked the stats of the docker container it's hardly utilising the resources only consuming CPU, no memory usage at all, https://github.com/airbytehq/airbyte/issues/12532 I read on this issue that the connectors are slow because it traverse 1 records at one time, and prints it. What to do?? I need the performance of 20k-30k records per second.
Data Pipeline: From ETL to EL plus T
2 projects | dev.to | 8 Jan 2023
Yes, absolutely, Airbyte, and there are many similar solutions, but Airbyte is open source and relatively easy to use.
Show HN: Data integration platform with 300 open-source connectors
2 projects | news.ycombinator.com | 16 Dec 2022
Airbyte: Data integration platform with 300+ open-source connectors
2 projects | reddit.com/r/opensource | 16 Dec 2022
Just an advisory here, the (majority of the) shared platform itself is not Open Source. They provide an overview here.2 projects | reddit.com/r/opensource | 16 Dec 2022
What is data integration?
6 projects | dev.to | 9 Nov 2022
Cloud ETL Repo to Warehouse & Visualize Personal Data
2 projects | reddit.com/r/QuantifiedSelf | 28 Oct 2022
Transfer data in Fauna to your analytics tool using Airbyte
2 projects | dev.to | 25 Oct 2022
git clone https://github.com/airbytehq/airbyte.git cd airbyte docker-compose up2 projects | dev.to | 25 Oct 2022
We are excited to introduce Fauna’s new Airbyte open source connector. This connector lets you replicate Fauna data into your data warehouses, lakes, and analytical databases, such as Snowflake, Redshift, S3, and more.
What are some alternatives?
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
dagster - An orchestration platform for the development, production, and observation of data assets.
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
spark-rapids - Spark RAPIDS plugin - accelerate Apache Spark with GPUs
dbt-core - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
jitsu - Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
dbt - dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications. [Moved to: https://github.com/dbt-labs/dbt-core]
supabase - The open source Firebase alternative. Follow to stay updated about our public Beta.
superset - Apache Superset is a Data Visualization and Data Exploration Platform