Mining metrics from unstructured logs

• coroot-node-agent

  8 272 8.3 Go

  A Prometheus exporter based on eBPF that gathers comprehensive container metrics

  

At Coroot, we implemented an automated log parsing in our open-source Prometheus exporter node-agent. To explain how it works, let's follow the same steps as I mentioned above.

• klog

  2 509 6.2 Go

  Leveled execution logs for Go (fork of https://github.com/golang/glog)

  

I would say it's not actually detection, it's more like guessing. For most log formats, the agent simply looks for a mention of known levels at the beginning of each message. In addition, we implemented special parsers for glog/klog and redis log formats. If it is impossible to detect the level of severity, the UNKNOWN level is used.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• logparser

  1 17 6.2 Go

  A library and a CLI tool for clustering unstructured logs. (by coroot)

  

In the first step, the agent removes quoted parts, parts in brackets, HEX numbers, UUIDs, and numeric values. Then, the remaining words are used to calculate the message group fingerprint. We extracted the code related to log parsing to this separate repository along with a command-line tool that can parse a log stream from stdin.

• loghub

  5 1,524 5.3

  A large collection of system log datasets for AI-driven log analytics [ISSRE'23]

  

Here is a summary for a log from the logpai/loghub dataset (kudos to the Logpai team for sharing this dataset):

• coroot-aws-agent

  2 18 4.9 Go

  A prometheus exporter that gathers metrics from AWS services.

  

AWS-agent exports the similar metrics aws_rds_log_messages_total related to every discovered RDS instance.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project