Parsing multiline logs using a custom Fluent Bit configuration

• newrelic-fluent-bit-output

  1 22 5.6 Go

  A Fluent Bit output plugin that sends logs to New Relic

  

If you’re already using Fluent Bit, you can also forward your Kubernetes logs to New Relic with the help of our Fluent Bit output plugin. Alternatively you can use it as a standalone Docker image, which we refer to as our Kubernetes plugin.

• helm-charts

  1 93 9.2 Smarty

  Helm charts for New Relic applications (by newrelic)

  

To configure Fluent Bit within Helm, we need to make changes to the fluent-bit-config configmap to tell it to apply the parsing.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• fluentbit-containerd-cri-o-json-log

  1 91 0.0

  Parsing CRI JSON logs with Fluent Bit - applies to fluentbit, kubernetes, containerd and cri-o

  

Parser is mapped to the value of the LOG_PARSER environment variable defined in the New Relic logging daemonset. Ensure Parser is set to “CRI” for this test, because AKS uses containerd as the container runtime and its log format is CRI-Log.

• fluent-bit

  35 5,344 9.8 C

  Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows

  

apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: newrelic labels: k8s-app: newrelic-logging data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level ${LOG_LEVEL} Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE output-newrelic.conf @INCLUDE filter-kubernetes.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path ${PATH} Parser ${LOG_PARSER} DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name multiline Match * multiline.parser multiline-regex [FILTER] Name record_modifier Match * Record cluster_name ${CLUSTER_NAME} [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log Off output-newrelic.conf: | [OUTPUT] Name newrelic Match * licenseKey ${LICENSE_KEY} endpoint ${ENDPOINT} parsers.conf: | # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [MULTILINE_PARSER] name multiline-regex key_content message type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|--------------------------------|----------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"

• www.yaml.org

  27 315 0.0 HTML

  The yaml.org website

  

If you're unfamiliar, Fluent Bit is a logging and metrics processor and forwarder. The New Relic infrastructure agent is bundled with a Fluent Bit plugin, so you can natively forward logs with the simple configuration of a YAML file.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project