mtail
loki
Our great sponsors
mtail | loki | |
---|---|---|
23 | 79 | |
3,727 | 21,863 | |
1.2% | 3.4% | |
8.7 | 9.9 | |
8 days ago | 6 days ago | |
Go | Go | |
Apache License 2.0 | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mtail
-
i need to visualize all logs from remote dir
You can do that with something like mtail. Basically write expressions that match your logs and produce metrics.
-
Tool to scrape (semi)-structured log files (e.g. log4j)
mtail is a standard tool for this.
-
Free netflow collector that forwards messages to a syslog server?
I use goflow2 to do something like this. I don't specifically use syslog itself for this, but mtail to generate the metrics.
-
How to easily gather IPv6 VS IPv4 usage on a web server?
I can recommend mtail. Here is a good example nginx script.
-
open-source tools to monitor JSON logs for unexpected patterns?
Convert your logs to metrics with mtail.
-
Server metrics monitoring and reporting for centos?
For nginx, you'll need to setup a log parser like mtail because it doesn't really have much for metrics to begin with.
-
How do you monitor webserver traffic?
mtail is an option if you have a set number of URLs. You'll need to either scrape the data it provides with prometheus, or send it to collectd, statsd, graphite, etc.
-
Another System Admin is always finding "problems" with my services but when I ask him to show me what is not working, it's working fine. Getting really annoyed.
Same thing with metrics, use mtail to parse dhcp logs. This way you can see graphs of requests and errors. Plus you can now write alerts for it in Prometheus.
-
Faster Alternatives to Logwatch
I use mtail. In addition to the real-time mode, you can also run it in "one shot" mode to count matches.
-
How to monitor log for keywords and trigger email when found?
I'm a fan of mtail for extracting data from logs. This fits well with the overall Prometheus Monitoring monitoring that I use.
loki
-
List of your reverse proxied services
I also needed to make a small patch to Promtail to make this work: https://github.com/grafana/loki/pull/10256
-
About reading logs
We don't pull logs, we forward logs to a centralized logging service.
-
loki VS openobserve - a user suggested alternative
2 projects | 30 Aug 2023
-
Logs monitoring with Loki, Node.js and Fastify.js
Over the past few months, I've been spending a lot of time creating dashboards on Grafana using Loki for MyUnisoft (the company I work for).
-
OpenObserve: Open source Elasticsearch alternative in Rust for logs. 140x lower storage cost
For log systems you generally don't migrate data. Logs lose value over time. What you want to do is to go ahead and start ingesting data into the new system (OpenObserve in this case) and slowly, the data in the old system will become stale and then you can retire it. However if you need to export logs anyhow, there is no straightforward way in loki to do this. You could run a script to query loki and export it to a file. If found this thread with a sample script - https://github.com/grafana/loki/issues/409
-
i need to visualize all logs from remote dir
Loki
-
I can't recommend serious use of an all-in-one local Grafana Loki setup
If you use Loki as the default logging driver with Docker and the Loki container shuts down, the rest of your containers will freeze up. This has been an issue for almost 3 years.
I installed promtail a few weeks back and I ran into this bug, that has been outstanding for months: https://github.com/grafana/loki/issues/8663 (e.g. a fix had been written but had not been released):
Due to a buffering issue, Loki would exit in case of configuration error without printing any error message or anything at all
There is definitely something weird about how the project is run.
Loki docs are here: https://github.com/grafana/loki/tree/main/docs/sources they are OSS.
The last contributor to the docs was an hour ago (at time of writing this comment) and came from a maintainer not employed by Grafana Labs.
Looking down the recent commits I see lots of activities from non-Grafana employees that have been accepted.
If there are specific issues with contributing docs or code please do point me towards them.
-
Tool to scrape (semi)-structured log files (e.g. log4j)
There are also log forwarding tools like promtail and fluentbit that can be used to both ship logs to something like Loki and produce metrics.
What are some alternatives?
ClickHouse - ClickHouse® is a free analytics DBMS for big data
fluent-bit - Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
Zabbix - Real-time monitoring of IT components and services, such as networks, servers, VMs, applications and the cloud.
VictoriaMetrics - VictoriaMetrics: fast, cost-effective monitoring solution and time series database
ElastiFlow - Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
loki-multi-tenant-proxy - Grafana Loki multi-tenant Proxy. Needed to deploy Grafana Loki in a multi-tenant way
oauth2-proxy - A reverse proxy that provides authentication with Google, Azure, OpenID Connect and many more identity providers.
cortex - A horizontally scalable, highly available, multi-tenant, long term Prometheus.
Typesense - Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences
agent - Vendor-neutral programmable observability pipelines.
alertmanager - Prometheus Alertmanager
rsyslog - a Rocket-fast SYStem for LOG processing