mulog VS logstash-logback-encoder

There are logging libraries that include syntactically scoped timers, such as mulog (https://github.com/BrunoBonacci/mulog). While a great library, we preferred timbre (https://github.com/taoensso/timbre) and rolled our own logging timer macro that interoperates with it. More convenient to have such niceties in a Lisp of course.

Assuming the crash doesn't cause the process to completely exit, you could indeed use `scope-capture` for this. This works well for local dev. In theory, you could use `sc.api/spy` in production code, and then attach a remote repl to diagnose any crashes. I wouldn't recommend this though, I think it would be best to use a good logging library like Mulog: https://github.com/BrunoBonacci/mulog

I use the logstash json format.

We're using https://github.com/logfellow/logstash-logback-encoder with logback (on Play Framework, but should work fine on Lambda as well).

For those interested in the details, I've created an example implementation based on Spring-flavoured REST and Logbook+logstash-logback-encoder within my own json-log-filter project for PoC / reference.

what's the difference to https://github.com/logfellow/logstash-logback-encoder ??

Don't use filebeat. Filebeat is for systems that you cannot change logging for. Push logs directly to logstash via logstash appender. Since I'm mainly logback user, there's one directly by logstash at https://github.com/logstash/logstash-logback-encoder. Quick search indicates that there's https://github.com/viskan/logstash-appender/ for log4j also and it seems it also supports MDC abuse as indicated by https://github.com/viskan/logstash-appender/blob/master/src/main/java/com/viskan/log4j/logstash/appender/LogstashAppender.java#L256. By abusing the MDC you won't need to write a processing pattern in logstash to extract metadata from giant blob line as each key in MDC will get assigned additional value, making your records in elastic search more useful.

You can use the Logstash Logback encooder. You mentioned Elk, so there must be a Logstash running somewhere you can connect to with this appender

We need to add traceId and spanId values to the application log. In production we would use the logstash-logback-encoder to generate logs in JSON format and send them to an ELK but for the demo we use this plain text logback layout: