PostgreSQL alternative for Large amounts of data

• timescaledb-parallel-copy

  2 348 4.5 Go

  A binary for parallel copying of CSV data into a TimescaleDB hypertable

  

As @dcalde mentioned before, apart from the timescaledb extension, there is parallel-copy (https://github.com/timescale/timescaledb-parallel-copy) which uses multiple threads / connections to bulk insert CSV files into postgresql. It works with timescale's hypertables, but also vanilla postgres tables.

• datafusion

  55 5,086 9.9 Rust

  Apache DataFusion SQL Query Engine

  

I've run datafusion over a collection of parquet files I had converted from csv. Was neat, but on a single VM it wasn't too hard with my dataset to do a group by query that blew through 64GiB ram.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• pgloader

  31 5,078 3.0 Common Lisp

  Migrate to PostgreSQL in a single command!

• rows

  1 860 5.1 Python

  A common, beautiful interface to tabular data, no matter the format

I have used following python utility https://github.com/turicas/rows , it provides cli that can bulk load csv to postgres tables.

• bellboy

  3 93 6.3 TypeScript

  Highly performant JavaScript data stream ETL engine.

  

If you are doing processing on each row, I would suggest that take a look at this library https://github.com/Claviz/bellboy It’s leveraging node js streams and provides quite a few essentials required for such etl processing.

• etlp

  1 3 5.7 Clojure

  A transducers based etl processing library written in clojure

You can take a look into code if you want to, but it’s heavily wip right now https://github.com/aregee/etlp

• q

  46 10,126 2.1 Python

  q - Run SQL directly on delimited files and multi-file sqlite databases (by harelba)

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project