-
hyperdx
Resolve production issues, fast. An open source observability platform unifying session replays, logs, metrics, traces and errors powered by Clickhouse and OpenTelemetry.
-
cypress-recorder
A Cypress plugin that generates test scripts from your interactions, a replacement Cypress Studio for Cypress v10 🖱 ⌨
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
-
-
highlight
highlight.io: The open source, full-stack monitoring platform. Error monitoring, session replay, logging, distributed tracing, and more.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
openobserve
🚀 10x easier, 🚀 140x lower storage cost, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces, RUM, Error tracking, Session replay).
-
First paragraph https://github.com/hyperdxio/hyperdx
"HyperDX helps engineers figure out why production is broken faster by centralizing and correlating logs, metrics, traces, exceptions and session replays in one place. An open source and developer-friendly alternative to Datadog and New Relic."
Perfect.
As merc, I never understood the why of Datadog (or equiv). The most common strategy of the teams and projects I rotated thru was "LOG ALL THE THINGS!" No purpose, no esthetics. General agreement about need to improve signal to noise ratio. But little courage or gumption. Such efforts would be easily rebuffed by citing the fable of Chesterfordstorm's Fences and something something velocity.
Late last century, IT projects, CRMs, ERPs, etc., were plagued by over collection of data. Opaque provenance, dubious (data) quality, unclear ownership, subtractive value propositions. No, no, don't remove that field. We might need it some day.
These "analytics" projects are the same, right? Every drive-by stakeholder tosses in a few tags, some misc fields, a little extra meta. And before anyone can say "kanban", the stone soup accreted enough mass to because it's own gravity well threatening implosion dragging the entire-org chart into the gapping maw of our universe's newest black hole.
Am I wrong?
But logging is useful, right? Or at least has that potential.
Does any one talk or write about this? (Those SRE themed novels are now buried deep in my reading list.)
I'd like some cookbooks or blue prints which show some idealized logging strategies, with depictions of common enough troubleshooting scenarios.
Having something authoritative to cite would make cranks like me look less like Eeyores. "Hey, team mates, you know what'd be really great?! Correlation IDs! So we can see how an action percolates thru our system!"
Just curious.
It's on the front page of the app, the company behind this (DeploySentinel) is YC backed: https://www.crunchbase.com/organization/deploysentinel. The original product seems like some kind of CI tool.
Interestingly, it seems like HyperDX might've been part of their original product offering that they decided to open source--their main website (https://www.deploysentinel.com) doesn't include any references to "HyperDX for CI" in May of 2023: https://web.archive.org/web/20230321102146/https://www.deplo...
Seems like they're pivoting to metrics? Even more of a reason to be weary about this.
This could be huge for healthcare companies like Metriport [1] - do you sign BAAs with customers for HIPAA compliance?
[1] https://github.com/metriport/metriport
Potentially useful resource – https://github.com/gajus/liqe
[2] https://github.com/highlight/highlight/tree/main
Ah yeah the easiest way is probably using the OpenTelemetry collector to set up a process to pull your logs out of jounrnald and send them via otel logs to HyperDX (or anywhere else that speaks otel) - the docs might be a bit tricky to go around depending on your familiarity with OpenTelemetry but this is what you'd be looking for:
https://github.com/open-telemetry/opentelemetry-collector-co...
Happy to dive more into the discord too if you'd like!
Ah yeah the easiest way is probably using the OpenTelemetry collector to set up a process to pull your logs out of jounrnald and send them via otel logs to HyperDX (or anywhere else that speaks otel) - the docs might be a bit tricky to go around depending on your familiarity with OpenTelemetry but this is what you'd be looking for:
https://github.com/open-telemetry/opentelemetry-collector-co...
Happy to dive more into the discord too if you'd like!
A good one. A lot is being built on top of clickhouse. I can count at least 3 if not more (hyperdx, signoz and highlight) built on top of clickhouse now.
We at OpenObserve are solving the same problem but a bit differently. A much simpler solution that anyone can run using a single binary on their own laptop or in a cluster of hundreds of nodes backed by s3. Covers logs, metrics, traces, Session replay, RUM and error tracking are being released by end of the month) - https://github.com/openobserve/openobserve
PS- I just skimmed https://opentelemetry.io, which your readme.md links to.
Good stuff. Much industry progress since I was last in the arena.
Their site has words about manual and automatic instrumentation. I'd have to dig a bit to see what they mean.
--
So. Remembering a bit more... Will try to keep this brief; you're a busy person.
> tend to log useless information or fail to tag them in ways that are actually searchable
#1 - I don't know know to manage lifecycle of meta. Who needs what? When is it safe to remove stuff?
We logged a lot of URLs. So many URL params. And when that wasn't crazy enough, over flow into HTTP headers. Plus partially duplicate, incorrectly, info in the payloads, a la SOAP. ("A person with two watches has no idea what time it is.")
When individual teams were uncertain, they'd just forward everything they received (copypasta), and add their own stuff.
Just add correlation IDs, right?
Ah, but there's "legacy". And unsupported protocols, like Redis and JDBC. And brain dead 3rd party services, with their own brain dead CSRs and engrs.
This is really dumb, and just propagates badness, but a few times, in a pinch, I've created Q&D "logging proxy". Just to get some visibility. So dumb.
Why stop there? Just have "the fabric" record stuff. Repurpose Wireguard into an Omni Logger. (Like the NSA does. Probably.) That'd eliminate most I/O trace style logging, right?
Image all these "webservices" and serverless apps without any need for instrumentation. Just have old school app level logging.
#2 - So much text processing.
An egregious example is logging HTTP headers. Serialize them as JSON and send that payload to a logging service. Which then rehydrate and store it some where.
My radical idea, which exactly no one has bought into, is to just pipe HTTP (Requests and Responses) as-is to log files. Then rotate, groom, archive, forward, inject, compress, whatever as desired.
That's what I did on the system I mentioned. All I/O was just streamed to files. And in the case of the HL7 (medical records stuff), it was super easy to extract the good bits, use that for Lucene's metadata, and store the whole message as the Lucene document.
I know such a radical idea is out of scope for your work. Just something fun to think about.
#3
> if none of your logs let you link service performance to customer X
Yup. Just keep adding servers. Kick the can down the road.
One team I helped had stuff randomly peg P95. And then sometimes a seemingly unrelated server would tip over. Between timeouts, retries, and load balancers, it really seemed like the ankle bone was connected to the shoulder bone. It just made no sense.
Fortunately, I had some prior experience. Being new to nodejs, maybe 5 years ago, I was shocked to learn there was no notion of back pressure. It was a challenging concept to explain to those teammates. But the omission of backpressure, and a hunch, was a good place for me start. (I'm no Dan Luu or Bryan Cantrill.)
I'd like to think that proper end-to-end logging, and the ability to find signal in the noise, diagnosis would have been more mundane.