rsyslog
fluent-bit
Our great sponsors
rsyslog | fluent-bit | |
---|---|---|
14 | 35 | |
1,953 | 5,344 | |
1.4% | 2.9% | |
9.1 | 9.8 | |
11 days ago | 3 days ago | |
C | C | |
GNU Lesser General Public License v3.0 only | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rsyslog
- My collection of Ansible roles for self-hosting everything with Rocky Linux and FreeIPA
-
Reason for random shutdown?
grep -iv ': starting\|kernel: .*: Power Button\|watching system buttons\|Stopped Cleaning Up\|Started Crash recovery kernel' \ /var/log/messages /var/log/syslog /var/log/apcupsd* \ | grep -iw 'recover[a-z]*\|power[a-z]*\|shut[a-z ]*down\|rsyslogd\|ups' /var/log/syslog:Apr 4 10:11:57 Asus-Mint kernel: [ 211.183400] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 4 10:12:57 Asus-Mint kernel: [ 271.484057] intel_powerclamp: Start idle injection to reduce power [i][b]/var/log/syslog:Apr 5 09:16:53 Asus-Mint apparmor.systemd[559]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Finished Create final runtime dir for shutdown pivot root. /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Finished Record System Boot/Shutdown in UTMP. /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: rsyslogd's groupid changed to 110 /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: rsyslogd's userid changed to 104 /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="651" x-info="https://www.rsyslog.com"] start /var/log/syslog:Apr 5 09:16:53 Asus-Mint kernel: [ 5.027861] systemd-journald[284]: File /var/log/journal/b76cc7b1bbdc489e93909d2043031de8/system.journal corrupted or uncleanly shut down, renaming and replacing. /var/log/syslog:Apr 5 09:16:53 Asus-Mint kernel: [ 6.129993] ath: phy0: Disable PLL PowerSave /var/log/syslog:Apr 5 09:16:53 Asus-Mint dbus-daemon[635]: dbus[635]: Unknown group "power" in message bus configuration file /var/log/syslog:Apr 5 09:16:53 Asus-Mint NetworkManager[636]: [1680679013.6224] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-dns-resolved.conf, 20-connectivity-ubuntu.conf, no-mac-addr-change.conf) (run: 10-globally-managed-devices.conf) (etc: default-wifi-powersave-on.conf) /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Started Unattended Upgrades Shutdown. /var/log/syslog:Apr 5 09:16:55 Asus-Mint kernel: [ 9.731489] wlp3s0: Limiting TX power to 20 (20 - 0) dBm as advertised by b0:5d:d4:81:19:81 /var/log/syslog:Apr 5 09:16:57 Asus-Mint kernel: [ 12.119484] systemd-journald[284]: File /var/log/journal/b76cc7b1bbdc489e93909d2043031de8/user-1000.journal corrupted or uncleanly shut down, renaming and replacing. /var/log/syslog:Apr 5 09:17:02 Asus-Mint dbus-daemon[635]: [system] Activating via systemd: service name='org.freedesktop.UPower' unit='upower.service' requested by ':1.45' (uid=1000 pid=1805 comm="csd-power " label="unconfined")[/b][/i] /var/log/syslog:Apr 5 09:17:03 Asus-Mint systemd[1]: Started Daemon for power management. /var/log/syslog:Apr 5 09:26:32 Asus-Mint kernel: [ 587.458982] powercap intel-rapl:0: package locked by BIOS, monitoring only /var/log/syslog:Apr 5 09:26:37 Asus-Mint kernel: [ 592.464754] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:43 Asus-Mint kernel: [ 598.472110] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:49 Asus-Mint kernel: [ 604.478361] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:51 Asus-Mint kernel: [ 606.480332] intel_powerclamp: Start idle injection to reduce power
-
Distributed syslog servers
Found this that might do what you want. https://github.com/rsyslog/rsyslog/issues/3913 It explains using omudpspoof an rsyslog modules that can spoof the senders IP to from the forwarding server. https://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html
- Help with freezing problem
-
Sending UDM system logs to rsyslog
I'm having trouble setting up a Dream Machine (non-pro) to send system logs to a machine on my network running rsyslog.
-
Help me find a better rsyslog template
%HOSTNAME%,) that does what I want, but I can not find any place where the macros are listed. Not the man page nor the documentation at https://www.rsyslog.com/. Apparently this is a legacy template format and there is a new and improved format. I think I'll settle for what I've got rather than take the days or weeks it would take to figure that out.
-
My logs ahow I'm being scanned by this, how do I set a firewall rule to drop the IP
May 1 00:05:17 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:17 debianTim systemd[1]: logrotate.service: Deactivated successfully.May 1 00:05:17 debianTim systemd[1]: Finished Rotate log files.May 1 00:05:17 debianTim audisp-syslog: type=SERVICE_START msg=audit(1651381517.328:5109): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"May 1 00:05:17 debianTim systemd[1]: logrotate.service: Consumed 4.498s CPU time.May 1 00:05:17 debianTim audisp-syslog: type=SERVICE_STOP msg=audit(1651381517.328:5110): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"May 1 00:05:17 debianTim audisp-syslog: type=BPF msg=audit(1651381517.332:5111): prog-id=0 op=UNLOADMay 1 00:05:17 debianTim rsyslogd: [origin software="rsyslogd" swVersion="8.2204.0" x-pid="865" x-info="https://www.rsyslog.com"] rsyslogd was HUPedMay 1 00:05:17 debianTim kernel: [34508.585138] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:18 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:19 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:20 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:20 debianTim kernel: [34511.587589] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:21 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:23 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:23 debianTim kernel: [34514.591944] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:24 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:26 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:26 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:26 debianTim kernel: [34517.594684] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:27 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:29 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:29 debianTim kernel: [34520.597010] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:30 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:32 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:32 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:32 debianTim kernel: [34523.599521] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:33 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:34 debianTim kernel: [34524.887982] FW REJECT (input): IN=enp1s0 OUT= MAC=01:00:5e:00:00:fb:1a:46:f9:6a:58:1d:08:00 SRC=192.168.0.172 DST=224.0.0.251 LEN=105 TOS=0x00 PREC=0x00 TTL=255 ID=34594 DF PROTO=UDP SPT=5353 DPT=5353 LEN=85 May 1 00:05:35 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:35 debianTim kernel: [34526.602159] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:36 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:37 debianTim eddie-ui.desktop[4669]: . 2022.05.01 00:05:37 - Collect information about AirVPN completedMay 1 00:05:38 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:38 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:38 debianTim kernel: [34529.604182] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:39 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:41 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:41 debianTim kernel: [34532.606391] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:42 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31
- Syslog server
- Forward firewalla logs
-
Linus Mint crashed completely. Not sure what is the solution
Jan 06 00:00:05 Eluktronics-Max-17 systemd[1]: Started CUPS Scheduler. Jan 06 00:00:05 Eluktronics-Max-17 systemd[1]: Started Make remote CUPS printers available locally. Jan 06 00:00:06 Eluktronics-Max-17 rsyslogd[814]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="814" x-info="https://www.rsyslog.com"] rsyslogd was HUPed Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: logrotate.service: Succeeded. Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: Finished Rotate log files. Jan 06 00:00:06 Eluktronics-Max-17 audit[39592]: AVC apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=39592 comm="cups-browsed" capability=23 capname="sys_nice" Jan 06 00:00:06 Eluktronics-Max-17 kernel: kauditd_printk_skb: 13 callbacks suppressed Jan 06 00:00:06 Eluktronics-Max-17 kernel: audit: type=1400 audit(1641445206.109:25): apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=39592 comm="cups-browsed" capability=23 capname="sys_nice" Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: man-db.service: Succeeded. Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: Finished Daily man-db regeneration. Jan 06 00:00:07 Eluktronics-Max-17 colord[1173]: failed to get session [pid 39591]: No data available Jan 06 00:00:10 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:00:21 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:01:24 Eluktronics-Max-17 kernel: mce_notify_irq: 4 callbacks suppressed Jan 06 00:01:24 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:01:37 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:02:54 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:02:54 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:03:04 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:04:07 Eluktronics-Max-17 kernel: mce_notify_irq: 2 callbacks suppressed Jan 06 00:04:07 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:04:24 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:05:11 Eluktronics-Max-17 kernel: mce_notify_irq: 2 callbacks suppressed Jan 06 00:05:11 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:05:35 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:06:29 Eluktronics-Max-17 kernel: mce_notify_irq: 1 callbacks suppressed Jan 06 00:06:29 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:06:33 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:07:33 Eluktronics-Max-17 kernel: mce_notify_irq: 4 callbacks suppressed Jan 06 00:07:33 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:07:34 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:08:38 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:08:38 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:08:53 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:09:58 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:09:58 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:10:28 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged -- Reboot --
fluent-bit
-
Observability at KubeCon + CloudNativeCon Europe 2024 in Paris
Fluentbit
-
Fluent Bit with ECS: Configuration Tips and Tricks
$ docker run --rm fluent-bit-dummy WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Fluent Bit v1.9.10 * Copyright (C) 2015-2022 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io [2023/12/24 16:06:59] [ info] [fluent bit] version=1.9.10, commit=557c8336e7, pid=1 [2023/12/24 16:06:59] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128 [2023/12/24 16:06:59] [ info] [cmetrics] version=0.3.7 [2023/12/24 16:06:59] [ info] [output:stdout:stdout.0] worker #0 started [2023/12/24 16:06:59] [ info] [sp] stream processor started [0] dummy.0: [1703434019.553880465, {"message"=>"custom dummy"}] [0] dummy.0: [1703434020.555768799, {"message"=>"custom dummy"}] [0] dummy.0: [1703434021.550525174, {"message"=>"custom dummy"}] [0] dummy.0: [1703434022.551563050, {"message"=>"custom dummy"}] [0] dummy.0: [1703434023.551944509, {"message"=>"custom dummy"}] [0] dummy.0: [1703434024.550027843, {"message"=>"custom dummy"}] [0] dummy.0: [1703434025.550901801, {"message"=>"custom dummy"}] [0] dummy.0: [1703434026.549279385, {"message"=>"custom dummy"}] ^C[2023/12/24 16:07:08] [engine] caught signal (SIGINT) [0] dummy.0: [1703434027.549678344, {"message"=>"custom dummy"}] [2023/12/24 16:07:08] [ warn] [engine] service will shutdown in max 5 seconds [2023/12/24 16:07:08] [ info] [engine] service has stopped (0 pending tasks) [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopping... [2023/12/24 16:07:08] [ info] [output:stdout:stdout.0] thread worker #0 stopped
-
Should You Be Scared of Unix Signals?
> Libc is a lot more tricky about signals, since not all libc functions can be safely called from handlers.
And this is a huge thing. People do all kinds of operations in signal handlers completely oblivious to the pitfalls. Pitfalls which often do not manifest, making it a great "it works for me" territory.
I once raised a ticket on fluentbit[1] about it but they have abused signal handlers so thoroughly that I do not think they can mitigate the issue without a major rewriting of the signal and crash handling.
[1] https://github.com/fluent/fluent-bit/issues/4836
-
Vector: a Rust-based lightweight alternative to Fluentd/Logstash
Fluentbit is Fluentd's lightweight alternative to itself.
https://fluentbit.io
- FLaNK Stack Weekly for 14 Aug 2023
-
Ultimate EKS Baseline Cluster: Part 1 - Provision EKS
From here, we can explore other developments and tutorials on Kubernetes, such as o11y or observability (PLG, ELK, ELF, TICK, Jaeger, Pyroscope), service mesh (Linkerd, Istio, NSM, Consul Connect, Cillium), and progressive delivery (ArgoCD, FluxCD, Spinnaker).
-
Fluentbit Kubernetes - How to extract fields from existing logs
From this (https://github.com/fluent/fluent-bit/issues/723), I can see there is no grok support for fluent-bit.
-
Parsing multiline logs using a custom Fluent Bit configuration
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: newrelic labels: k8s-app: newrelic-logging data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level ${LOG_LEVEL} Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE output-newrelic.conf @INCLUDE filter-kubernetes.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path ${PATH} Parser ${LOG_PARSER} DB /var/log/flb_kube.db Mem_Buf_Limit 7MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name multiline Match * multiline.parser multiline-regex [FILTER] Name record_modifier Match * Record cluster_name ${CLUSTER_NAME} [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log Off output-newrelic.conf: | [OUTPUT] Name newrelic Match * licenseKey ${LICENSE_KEY} endpoint ${ENDPOINT} parsers.conf: | # Relevant parsers retrieved from: https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name cri Format regex Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]*) (?.*)$ Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z [MULTILINE_PARSER] name multiline-regex key_content message type regex flush_timeout 1000 # # Regex rules for multiline parsing # --------------------------------- # # configuration hints: # # - first state always has the name: start_state # - every field in the rule must be inside double quotes # # rules | state name | regex pattern | next state # ------|---------------|--------------------------------|----------- rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont" rule "cont" "/^\s+at.*/" "cont"
-
Tool to scrape (semi)-structured log files (e.g. log4j)
There are also log forwarding tools like promtail and fluentbit that can be used to both ship logs to something like Loki and produce metrics.
-
How to Deploy and Scale Strapi on a Kubernetes Cluster 2/2
FluentBit, is a logging processor that can help you to push all of your application logs to a central location like an ElasticSearch or OpenSearch cluster.
What are some alternatives?
syslog-ng - syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
loki - Like Prometheus, but for logs.
librdkafka - The Apache Kafka C/C++ library
jaeger - CNCF Jaeger, a Distributed Tracing Platform
Flume - Mirror of Apache Flume
winston - A logger for just about everything.
lua-mongo - MongoDB Driver for Lua
Grafana - The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
FreeRADIUS - FreeRADIUS - A multi-protocol policy server.
stanza - Fast and lightweight log transport and processing.