Top 21 data-transformation Open-Source Projects

• prql

  106 9,427 9.9 Rust

  PRQL is a modern language for transforming data — a simple, powerful, pipelined SQL replacement

  

Project mention: Prolog language for PostgreSQL proof of concept | news.ycombinator.com | 2024-03-30

• Mage

  77 7,001 9.9 Python

  🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

  

Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• glom

  2 1,825 7.4 Python

  ☄️ Python's nested data operator (and CLI), for all your declarative restructuring needs. Got data? Glom it! ☄️

Project mention: Ask HN: How can I get better at writing production-level Python? | news.ycombinator.com | 2023-07-18

• Optimus

  0 1,446 1.9 Python

  :truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)

  

• pglogical

  2 935 3.9 C

  Logical Replication extension for PostgreSQL 15, 14, 13, 12, 11, 10, 9.6, 9.5, 9.4 (Postgres), providing much faster replication than Slony, Bucardo or Londiste, as well as cross-version upgrades.

  

• zingg

  23 877 9.3 Java

  Scalable identity resolution, entity resolution, data mastering and deduplication using ML

  

• optimus

  5 737 4.4 Go

  Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. (by raystack)

  

Project mention: Data Engineering Tools in Go | /r/dataengineering | 2023-05-18

You can check odpf github, they created some dataops tools using go, one of the example is optimus (https://github.com/odpf/optimus) which is a data pipeline orchestrator

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• prose

  4 612 7.2 C#

  Microsoft Program Synthesis using Examples SDK is a framework of technologies for the automatic generation of programs from input-output examples. This repo includes samples and sample data for the Microsoft Program Synthesis using Example SDK. (by microsoft)

  

• Porter

  1 608 2.7 PHP

  :lipstick: Durable and asynchronous data imports for consuming data at scale and publishing testable SDKs. (by ScriptFUSION)

  

• collapse

  2 599 9.6 C

  Advanced and Fast Data Transformation in R (by SebKrantz)

• sqawk

  1 308 10.0 Tcl

  Like awk but with SQL and table joins

• fastverse

  3 213 6.8 R

  An Extensible Suite of High-Performance and Low-Dependency Packages for Statistical Computing and Data Manipulation in R

  

• clojure-dsl-resources

  6 170 0.0

  A curated list of Clojure resources for dealing with domain-specific languages.

Project mention: Let's write a simple microservice in Clojure | dev.to | 2024-04-26

Compojure's DSL for web applications makes it easy to set up REST API routes with corresponding HTTP methods. Adding a Swagger API descriptor through libraries like ring-swagger provides a visual interface for interacting with the API and enables client code generation. You can use the Prismatic schema library for HTTP request validation and data coercing to ensure the API consumes and produces data that conforms to predefined schemas. Compojure's middleware approach allows for modular and reusable components that can handle cross-cutting concerns like authentication, logging, and request/response transformations, enhancing the API's scalability and maintainability.

• cq

  1 152 0.0 Clojure

  Clojure Command-line Data Processor for JSON, YAML, EDN, XML and more (by markus-wa)

• daany

  1 54 6.9 C#

  Daany - .NET DAta ANalYtics .NET library with the implementation of DataFrame, Time series decompositions and Linear Algebra routines BLASS and LAPACK.

• data-lens

  2 30 6.2 Common Lisp

  Functional utilities for Common Lisp

• Unquery

  10 16 3.1 C++

  Command line query tool for JSON files

  

• fragments

  2 14 2.8 TypeScript

  Transform and compose data for HTTP transactions.

  

• jsonata-playbook

  1 5 1.8 Go

  practical examples of jsonata [go-jsonata 1.5.4]

  

• cuphic

  2 4 0.0 Clojure

  Transform or scrape Hiccup with a declarative DSL.

  

• sqlite-wf

  1 2 0.0 C

  Simple visual ETL tool

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

data-transformation related posts

• SQL is syntactic sugar for relational algebra
  2 projects | news.ycombinator.com | 23 Mar 2024
• Anyone using PRQL over SQL in Production?
  1 project | /r/dataengineering | 6 Dec 2023
• SQL 😎😎😎
  1 project | /r/SQL | 12 Jun 2023
• Machine Learning, Linear Algebra, and More: Is SQL All You Need?
  2 projects | /r/ProgrammingLanguages | 20 May 2023
• Is there a "way to add or subtract x months from a date, that works in both PostgreSQL and SQL Server?
  1 project | /r/SQL | 7 May 2023
• Mage Battlegrounds: Craft insights from real-time customer behavior analysis
  2 projects | dev.to | 10 Apr 2023
• VAST 3.0 released. Open-Source Security Data Pipelines with Kusto-like syntax
  2 projects | /r/cybersecurity | 15 Mar 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 28 Apr 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com