Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Don't use filebeat. Filebeat is for systems that you cannot change logging for. Push logs directly to logstash via logstash appender. Since I'm mainly logback user, there's one directly by logstash at https://github.com/logstash/logstash-logback-encoder. Quick search indicates that there's https://github.com/viskan/logstash-appender/ for log4j also and it seems it also supports MDC abuse as indicated by https://github.com/viskan/logstash-appender/blob/master/src/main/java/com/viskan/log4j/logstash/appender/LogstashAppender.java#L256. By abusing the MDC you won't need to write a processing pattern in logstash to extract metadata from giant blob line as each key in MDC will get assigned additional value, making your records in elastic search more useful.
Don't use filebeat. Filebeat is for systems that you cannot change logging for. Push logs directly to logstash via logstash appender. Since I'm mainly logback user, there's one directly by logstash at https://github.com/logstash/logstash-logback-encoder. Quick search indicates that there's https://github.com/viskan/logstash-appender/ for log4j also and it seems it also supports MDC abuse as indicated by https://github.com/viskan/logstash-appender/blob/master/src/main/java/com/viskan/log4j/logstash/appender/LogstashAppender.java#L256. By abusing the MDC you won't need to write a processing pattern in logstash to extract metadata from giant blob line as each key in MDC will get assigned additional value, making your records in elastic search more useful.