vanus
openmeter
vanus | openmeter | |
---|---|---|
33 | 11 | |
2,262 | 890 | |
0.2% | 5.4% | |
7.2 | 9.8 | |
about 2 months ago | 9 days ago | |
Go | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
vanus
-
Open Source contribution and online communities !
https://github.com/linkall-labs/vanus would be a good choice.
-
Kafka ETL tool, is there any?
Hi, I think vanus can meet your needs very well. It is a message queue with message processing capabilities, including filter,transformation and etc. https://github.com/linkall-labs/vanus
-
Vanus: An open-source message queue with built-in event-processing capabilities
GitHub: https://github.com/linkall-labs/vanus
Vanus helps users build event pipelines between SaaS, cloud services, and cloud functions in minutes. https://github.com/linkall-labs/vanus Build the event-driven system Get events from cloud services and SaaS, and deliver them to cloud functions or microservices. Synchronize changed data or transfer data to the data lake. Obtain events generated by SaaS and send them to other SaaS. Out-of-the-box event computing capabilities Real-time processing during event transmissions, such as filtering and transformation. Natively support the CloudEvents specification, and can directly send events to workloads that support CloudEvent. 100% open source, Super easy to use One-click deployment, the installation is completed within 1 minute, and developers without MQ experience can also use it. Message queues and connectors are 100% open source, a one-stop open-source solution. Please share your thoughts and questions in the comments. I will be happy to answer each one of them.
- Vanus: An open-source message queue with event filtering and transformation capabilities
-
Vanus: An open-source message queue with event filtering and transformation capabilities discussion
GitHub: https://github.com/linkall-labs/vanus
openmeter
- OpenMeter – open-source Realtime Metering
- Ask HN: Which JavaScript lib for metering APIs should I use?
- Looking for feedback on our website for an Open Source project
-
ClickHouse for usage metering with Kafka Connect
This plugin guarantees exactly-once delivery between Kafka topics and ClickHouse tables, which is critical, as Kafka Connect tasks are only aware of the latest topic offset acknowledged by the consumer. For example, consumers can fail to acknowledge a processed offset due to a network error or an exception. This is great as exactly-once inserts prevent dropping or double-inserting usage, leading to incorrect billing.
In OpenMeter, we pre-aggregate usage events into one-minute tumbling windows to reduce the number of rows we need to scan at query time. To do this, with ClickHouse, we use the AggregatingMergeTree table engine that enables incremental data aggregation when combined with MaterializedView. In ClickHouse, MaterializedViews are trigger-based and update when new records are inserted into the source table. Consequently, the corresponding materialized views are updated whenever Kafka Connect transfers a batch of events to ClickHouse. This also means inserts can fail when the view cannot process a record at trigger. We send failed events into the Dead Letter Queue topic for later processing.
To help ClickHouse with hot topics, we will consider adding an extra streaming aggregation step for high-producers, but this time with a more horizontally scalable stream processor like Arroyo. This would reduce ClickHouse insert batch sizes. Based on our tests, ClickHouse works best if batch sizes are 50-100k and less frequent than per second.
To see it in action, check out our open-source repo: https://github.com/openmeterio/openmeter
- Real-Time and Scalable Usage Metering
- GitHub - openmeterio/openmeter: Accurate and real-time usage metering for AI, DevOps, billing and analytics.
-
How to meter POD execution duration for billing?
I’m researching how to meter a POD execution duration for auditable use cases like billing. We are working on an infrastructure metering solution, and our current example polls the k8s API every second to see which pods are running: see code here - wondering what folks think about this approach.
-
Real-Time, Open Source Usage Metering built on Kafka
OpenMeter leverages stream processing to update meters in real-time while simultaneously processing large volumes of events. The core is written in Go and uses the CloudEvents format to describe usage, Kafka to ingest events, and ksqlDB to dedupe and aggregate meters. We are also working on a Postgres sink for long-term storage. Check out our GitHub to learn more: https://github.com/openmeterio/openmeter
- Launch HN: OpenMeter (YC W23) – Real-Time, Open Source Usage Metering
What are some alternatives?
kubevela - The Modern Application Platform. [Moved to: https://github.com/kubevela/kubevela]
lago - Open Source Metering and Usage Based Billing API ⭐️ Consumption tracking, Subscription management, Pricing iterations, Payment orchestration & Revenue analytics
plumber - A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
kafka_manager - Simplifies eventing between microservices using kafka with kafka-go client
dapr - Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.
hop - Hop Orchestration Platform
event - 📢 Lightweight event manager and dispatcher implements by Go. Go实现的轻量级的事件管理、调度程序库, 支持设置监听器的优先级, 支持使用通配符来进行一组事件的监听
argo-events - Event-driven Automation Framework for Kubernetes
Faust - Python Stream Processing
kubevela - The Modern Application Platform.
faust - Python Stream Processing. A Faust fork
components-contrib - Community driven, reusable components for distributed apps