openmeter
kafka_manager
| openmeter | kafka_manager | |
|---|---|---|
| 11 | 1 | |
| 897 | 2 | |
| 6.1% | - | |
| 9.8 | 0.0 | |
| 6 days ago | almost 3 years ago | |
| Go | Go | |
| Apache License 2.0 | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
openmeter
- OpenMeter – open-source Realtime Metering
- Ask HN: Which JavaScript lib for metering APIs should I use?
- Looking for feedback on our website for an Open Source project
-
ClickHouse for usage metering with Kafka Connect
This plugin guarantees exactly-once delivery between Kafka topics and ClickHouse tables, which is critical, as Kafka Connect tasks are only aware of the latest topic offset acknowledged by the consumer. For example, consumers can fail to acknowledge a processed offset due to a network error or an exception. This is great as exactly-once inserts prevent dropping or double-inserting usage, leading to incorrect billing.
In OpenMeter, we pre-aggregate usage events into one-minute tumbling windows to reduce the number of rows we need to scan at query time. To do this, with ClickHouse, we use the AggregatingMergeTree table engine that enables incremental data aggregation when combined with MaterializedView. In ClickHouse, MaterializedViews are trigger-based and update when new records are inserted into the source table. Consequently, the corresponding materialized views are updated whenever Kafka Connect transfers a batch of events to ClickHouse. This also means inserts can fail when the view cannot process a record at trigger. We send failed events into the Dead Letter Queue topic for later processing.
To help ClickHouse with hot topics, we will consider adding an extra streaming aggregation step for high-producers, but this time with a more horizontally scalable stream processor like Arroyo. This would reduce ClickHouse insert batch sizes. Based on our tests, ClickHouse works best if batch sizes are 50-100k and less frequent than per second.
To see it in action, check out our open-source repo: https://github.com/openmeterio/openmeter
- Real-Time and Scalable Usage Metering
- GitHub - openmeterio/openmeter: Accurate and real-time usage metering for AI, DevOps, billing and analytics.
-
How to meter POD execution duration for billing?
I’m researching how to meter a POD execution duration for auditable use cases like billing. We are working on an infrastructure metering solution, and our current example polls the k8s API every second to see which pods are running: see code here - wondering what folks think about this approach.
-
Real-Time, Open Source Usage Metering built on Kafka
OpenMeter leverages stream processing to update meters in real-time while simultaneously processing large volumes of events. The core is written in Go and uses the CloudEvents format to describe usage, Kafka to ingest events, and ksqlDB to dedupe and aggregate meters. We are also working on a Postgres sink for long-term storage. Check out our GitHub to learn more: https://github.com/openmeterio/openmeter
- Launch HN: OpenMeter (YC W23) – Real-Time, Open Source Usage Metering
kafka_manager
What are some alternatives?
lago - Open Source Metering and Usage Based Billing API ⭐️ Consumption tracking, Subscription management, Pricing iterations, Payment orchestration & Revenue analytics
plumber - A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
vanus - Vanus is a Serverless, event streaming system with processing capabilities. It easily connects SaaS, Cloud Services, and Databases to help users build next-gen Event-driven Applications.
kafka-go - Kafka library in Go
goka - Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
kubeshark - The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes