How we scaled ingestion to one million rows per second

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • tsbs

    Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data

  • On the CrateDB side, the data model consists of a single table that stores CPU usage statistics from Unix-based operating systems. The data model was adopted from Timescale’s Time Series Benchmark Suite.

  • cratedb-terraform

    Terraform deployment configurations for CrateDB

  • crate-terraform: Terraform scripts to easily start CrateDB clusters in the cloud. For a certain set of variable values, it spins up a CrateDB cluster in a reproducible way (including a load balancer). It also allows configuring certain performance-critical properties, such as disk throughput. Going with Terraform guarantees that the setup will be easy to reproduce. We will run all infrastructure on the AWS cloud.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • nodeIngestBench

  • nodeIngestBench: The client tool generating batched INSERT statements. Implemented in Node.js, it provides the needed high concurrency with a pool of workers that run as separate child processes.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts