websocketd VS Bunyan

Somewhat related: https://github.com/joewalnes/websocketd

> websocketd is a small command-line tool that will wrap an existing command-line interface program, and allow it to be accessed via a WebSocket.

I hadn't even considered collecting traces/spans in this way yet, and have taken the approach of "stuff outputting logs in JSON format to stderr/local file". I usually end up writing a (temporary, structured) log message with the relevant span tags, but wouldn't it would be much better to run the actual trace/span code and be able to verify it locally without the ad-hoc log message?

The prototype I built is a web application that creates websocket connections, and if those connections receive messages that are JSON, log lines are added. Columns are built dynamically as log messages arrive, and then you can pick which columns to render in the table. If you're curious here's the code, including a screenshot: https://github.com/corytheboyd-smartsheet/json-log-explorer

With websockets, it's very easy to use websocketd (http://websocketd.com), which will watch input files for new lines, and write them verbatim as websocket messages to listeners (the web app).

To make the idea real, would want to figure out how to not require the user to run websocketd out of band, but watching good ol' files is dead simple, and very easy to add to most code (add a new log sink, use existing log file, etc.)

* Additionally, client can stream data to the backend server (if bidirectional GRPC streams are used). I.e. client sends WebSocket messages, those will be transformed to GRPC messages by WebSocket server and delivered to the application backend.

As a result we have a system which allows to quickly create individual streams by using strict GRPC contract but terminating connections over WebSocket transport. So it works well in web browsers. After that no need to write WebSocket protocol, client implementation, handle WebSocket connection. This all will be solved by a suggested WebSocket server and its client SDKs.

The mechanics is similar to Websocketd (https://github.com/joewalnes/websocketd), but instead of creating OS processes we create GRPC streams. The difference from grpc-web (https://github.com/grpc/grpc-web) is that we provide streaming capabilities but not exposing GRPC contract to the client - just allowing to stream any data as payload (both binary and text) with some wrappers from our client SDKs side for managing subscriptions. I.e. it's not native GRPC streams on the client side - we expose just Connection/Subscription object to stream in both directions. GRPC streams used only for communication between WebSocket server and backend. To mention - grpc-web does not support all kinds of streaming now (https://github.com/grpc/grpc-web#streaming-support) while proposed solution can. This all should provide a cross-platform way to quickly write streaming apps due to client SDKs and language-agnostic nature of GRPC.

I personally see both pros and cons in this scheme (without concentrating on both too much here to keep the question short). I spent some time thinking about this myself, already have some working prototypes – but turned out need more opinions before moving forward with the idea and releasing this, kinda lost in doubts.

My main question - whether this seems interesting for someone here? Do you find this useful and see practical value?

I also used to use this (http://websocketd.com/) along with netcat(1) before just biting the bullet and writing my own websocket library for our server as we needed to scale up slightly.

Another option if you’re ready to implement the frontend part is https://github.com/joewalnes/websocketd which has the advantage of streaming the output of your script

Wanted to add websocketd [1]. It's an amazing tool to stream debugging logs to another system where you can build your webapps that accumulate alerts.

Use it only for debugging builds and not for production (obviously).

[1] https://github.com/joewalnes/websocketd

NodeJS -> Pino, Winston, Bunyan, Npmlog, e.t.c.

Bunyan is also another popular and fast JSON Node.js logging library. Just like Winston, it also supports logging into multiple transport options. Other features include a neat-printing CLI for logs, a log filter, serializers for rendering objects, snooping system, and the ability to support multiple runtime environments such as NW.js and WebPack. Bunyan enforces the JSON format for logs.

bunyan argues that logs should be structured and that JSON is a good format for that. It describes itself as a “simple and fast JSON logging library” and has all the features you would expect from a logging library, including serializers and support for different runtime environments including Node.js, Browserify and Webpack.

Suppose that after some time you decide that the winston logger library was not the best logger for your project and you want to use Bunyan, what do you do? You just need to create a BunyanLogger class that implements the ILogger interface and it is ready to be used by the UserService.

Bunyan: Another popular logging library that outputs in JSON by default.

bunyan

Are there more options? Absolutely: Bunyan, Pino, and others. It depends on what your particular logging needs are.

With the increasing number of requests and possibly errors, we needed a proper logging setup to debug and monitor the service. That’s why we chose bunyan to log insightful data in our application. These logs sit conveniently on our new logging pipeline running on EFK (or, Elasticsearch Fluentd Kibana) stack. While this deserves a separate blog post on its own, let’s take a brief look at how the logs travel from our application to the kibana dashboard.

Bunyan — Another feature-rich logging framework that outputs in JSON by default and provides a CLI tool for viewing your logs.