Our great sponsors
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
I have been using airbyte sources, S3 mainly, it is so slow, I'm getting 1k-3k records per second, on a high end machine 4 Cpus and 16GB Ram. I checked the stats of the docker container it's hardly utilising the resources only consuming CPU, no memory usage at all, https://github.com/airbytehq/airbyte/issues/12532 I read on this issue that the connectors are slow because it traverse 1 records at one time, and prints it. What to do?? I need the performance of 20k-30k records per second.
-
Hi there! (Disclaimer: Im the founder of https://github.com/cloudquery/cloudquery) . We are an a high performance open source ELT framework written in Go.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.