Our great sponsors
-
manticoresearch
Easy to use open source fast database for search | Good alternative to Elasticsearch now | Drop-in replacement for E in the ELK soon
-
qryn
qryn is a polyglot, high-performance observability framework for ClickHouse. Ingest, store and analyze logs, metrics and telemetry traces from any agent supporting Loki, Prometheus, OTLP, Tempo, Elastic, InfluxDB and many more formats and query transparently using Grafana or any other compatible client.
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
-
-
signoz
SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
"The Coroner recorded every instruction executed, with its inputs and results, writing over the oldest record when it filled up. When a program crashed, you could punch out a full record of what it was doing in most of its last second and analyze the problem at your ease. I have often wished that PCs offered such an advanced feature."
So essentially buffer all logs into an in-memory circular buffer of capacity N. If a log record is emitted that has a certain severity/level; flush all records from the buffer to disk/clickHouse/grafana/whatever.
The python MemoryHandler[2] almost implements the said technique except that particular handler also flushes when buffer is full; which is not particularly what I would want.
1. https://authors.library.caltech.edu/5456/1/hrst.mit.edu/hrs/...
2. https://github.com/python/cpython/blob/v3.11.1/Lib/logging/h...
This sounds fairly similar to self-hosting SigNoz[1] except they use the OpenTelemetry Collector[2] in the place where the author has a custom Redis queue consumer.
1. https://signoz.io
2. https://opentelemetry.io/docs/collector/
This is awesome! We did something similar for our internal logging at Tinybird (which itself is built on ClickHouse) and I recently turned into it a (very simplified) Starter Kit that others can fork and use in their projects. https://github.com/tinybirdco/log-analytics-starter-kit
Rather than writing to files and using an agent like beats to tail them, it sends logs directly from the application code with a basic POST request. Obviously, you could just as happily tail the file and forward the logs, but this approach reduced the footprint of tools and made it work in serverless environments.