Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
But in terms of what you could do today, I'd actually suggest taking a look at cloudflare workers. You could have essentially the same design as described above, where you have an asynchronous job for each stage of the pipeline. For less than 1-2GB/month, a serverless architecture may be less expensive than an equivalent "streaming" option involving Kafka, Beam, etc because they shut down your processes when there's no data/requests to process. You could also do something equivalent with AWS lambda + kinesis or GCP cloud-run + pub-sub, but I just find cloudflare workers easier to work with.