syslog-ng VS rsyslog

Syslog-ng can feed splunk directly via HEC without having to use heavy forwarders. Earlier you had to hand configure it via the http () destination, v4.2 added this via a separate destination driver. https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.2.0

ports: syslog-ng 4.1.1[6]

Syslog-ng seems a bit weird about open/close source. They have a github but it seems like all the documentation is seemingly all over the place & it's not obvious to find what the limitations of open source version are.

What we do internally is use [syslog-ng](https://github.com/syslog-ng/syslog-ng) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

We're using syslog-ng in my apprenticeship, it works just fine, interfacing with a in-prem Prometheus/Elasticsearch

[12] https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.1

grep -iv ': starting\|kernel: .*: Power Button\|watching system buttons\|Stopped Cleaning Up\|Started Crash recovery kernel' \ /var/log/messages /var/log/syslog /var/log/apcupsd* \ | grep -iw 'recover[a-z]*\|power[a-z]*\|shut[a-z ]*down\|rsyslogd\|ups' /var/log/syslog:Apr 4 10:11:57 Asus-Mint kernel: [ 211.183400] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 4 10:12:57 Asus-Mint kernel: [ 271.484057] intel_powerclamp: Start idle injection to reduce power [i][b]/var/log/syslog:Apr 5 09:16:53 Asus-Mint apparmor.systemd[559]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Finished Create final runtime dir for shutdown pivot root. /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Finished Record System Boot/Shutdown in UTMP. /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: rsyslogd's groupid changed to 110 /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: rsyslogd's userid changed to 104 /var/log/syslog:Apr 5 09:16:53 Asus-Mint rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="651" x-info="https://www.rsyslog.com"] start /var/log/syslog:Apr 5 09:16:53 Asus-Mint kernel: [ 5.027861] systemd-journald[284]: File /var/log/journal/b76cc7b1bbdc489e93909d2043031de8/system.journal corrupted or uncleanly shut down, renaming and replacing. /var/log/syslog:Apr 5 09:16:53 Asus-Mint kernel: [ 6.129993] ath: phy0: Disable PLL PowerSave /var/log/syslog:Apr 5 09:16:53 Asus-Mint dbus-daemon[635]: dbus[635]: Unknown group "power" in message bus configuration file /var/log/syslog:Apr 5 09:16:53 Asus-Mint NetworkManager[636]: [1680679013.6224] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-dns-resolved.conf, 20-connectivity-ubuntu.conf, no-mac-addr-change.conf) (run: 10-globally-managed-devices.conf) (etc: default-wifi-powersave-on.conf) /var/log/syslog:Apr 5 09:16:53 Asus-Mint systemd[1]: Started Unattended Upgrades Shutdown. /var/log/syslog:Apr 5 09:16:55 Asus-Mint kernel: [ 9.731489] wlp3s0: Limiting TX power to 20 (20 - 0) dBm as advertised by b0:5d:d4:81:19:81 /var/log/syslog:Apr 5 09:16:57 Asus-Mint kernel: [ 12.119484] systemd-journald[284]: File /var/log/journal/b76cc7b1bbdc489e93909d2043031de8/user-1000.journal corrupted or uncleanly shut down, renaming and replacing. /var/log/syslog:Apr 5 09:17:02 Asus-Mint dbus-daemon[635]: [system] Activating via systemd: service name='org.freedesktop.UPower' unit='upower.service' requested by ':1.45' (uid=1000 pid=1805 comm="csd-power " label="unconfined")[/b][/i] /var/log/syslog:Apr 5 09:17:03 Asus-Mint systemd[1]: Started Daemon for power management. /var/log/syslog:Apr 5 09:26:32 Asus-Mint kernel: [ 587.458982] powercap intel-rapl:0: package locked by BIOS, monitoring only /var/log/syslog:Apr 5 09:26:37 Asus-Mint kernel: [ 592.464754] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:43 Asus-Mint kernel: [ 598.472110] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:49 Asus-Mint kernel: [ 604.478361] intel_powerclamp: Start idle injection to reduce power /var/log/syslog:Apr 5 09:26:51 Asus-Mint kernel: [ 606.480332] intel_powerclamp: Start idle injection to reduce power

Found this that might do what you want. https://github.com/rsyslog/rsyslog/issues/3913 It explains using omudpspoof an rsyslog modules that can spoof the senders IP to from the forwarding server. https://www.rsyslog.com/doc/v8-stable/configuration/modules/omudpspoof.html

I'm having trouble setting up a Dream Machine (non-pro) to send system logs to a machine on my network running rsyslog.

%HOSTNAME%,) that does what I want, but I can not find any place where the macros are listed. Not the man page nor the documentation at https://www.rsyslog.com/. Apparently this is a legacy template format and there is a new and improved format. I think I'll settle for what I've got rather than take the days or weeks it would take to figure that out.

May 1 00:05:17 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:17 debianTim systemd[1]: logrotate.service: Deactivated successfully.May 1 00:05:17 debianTim systemd[1]: Finished Rotate log files.May 1 00:05:17 debianTim audisp-syslog: type=SERVICE_START msg=audit(1651381517.328:5109): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"May 1 00:05:17 debianTim systemd[1]: logrotate.service: Consumed 4.498s CPU time.May 1 00:05:17 debianTim audisp-syslog: type=SERVICE_STOP msg=audit(1651381517.328:5110): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=logrotate comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"May 1 00:05:17 debianTim audisp-syslog: type=BPF msg=audit(1651381517.332:5111): prog-id=0 op=UNLOADMay 1 00:05:17 debianTim rsyslogd: [origin software="rsyslogd" swVersion="8.2204.0" x-pid="865" x-info="https://www.rsyslog.com"] rsyslogd was HUPedMay 1 00:05:17 debianTim kernel: [34508.585138] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:18 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:19 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:20 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:20 debianTim kernel: [34511.587589] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:21 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:23 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:23 debianTim kernel: [34514.591944] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:24 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:26 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:26 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:26 debianTim kernel: [34517.594684] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:27 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:29 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:29 debianTim kernel: [34520.597010] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:30 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:32 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:32 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:32 debianTim kernel: [34523.599521] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:33 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:34 debianTim kernel: [34524.887982] FW REJECT (input): IN=enp1s0 OUT= MAC=01:00:5e:00:00:fb:1a:46:f9:6a:58:1d:08:00 SRC=192.168.0.172 DST=224.0.0.251 LEN=105 TOS=0x00 PREC=0x00 TTL=255 ID=34594 DF PROTO=UDP SPT=5353 DPT=5353 LEN=85 May 1 00:05:35 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:35 debianTim kernel: [34526.602159] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:36 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:37 debianTim eddie-ui.desktop[4669]: . 2022.05.01 00:05:37 - Collect information about AirVPN completedMay 1 00:05:38 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:38 debianTim psad: scan detected fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 -> ff02:0000:0000:0000:0000:0000:0000:0001 DL: 3 total scan dsts: 1May 1 00:05:38 debianTim kernel: [34529.604182] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:39 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:41 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31May 1 00:05:41 debianTim kernel: [34532.606391] FW6 REJECT (input): IN=enp1s0 OUT= MAC=33:33:00:00:00:01:48:4e:fc:f0:69:b8:86:dd SRC=fe80:0000:0000:0000:4a4e:fcff:fef0:69b8 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=168 TC=0 HOPLIMIT=255 FLOWLBL=356592 PROTO=ICMPv6 TYPE=134 CODE=0 May 1 00:05:42 debianTim gnome-shell[3980]: JS ERROR: Gio.IOErrorEnum: Error opening file /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory#012cb@/usr/share/gnome-shell/extensions/[email protected]/extension.js:1614:31

Jan 06 00:00:05 Eluktronics-Max-17 systemd[1]: Started CUPS Scheduler. Jan 06 00:00:05 Eluktronics-Max-17 systemd[1]: Started Make remote CUPS printers available locally. Jan 06 00:00:06 Eluktronics-Max-17 rsyslogd[814]: [origin software="rsyslogd" swVersion="8.2001.0" x-pid="814" x-info="https://www.rsyslog.com"] rsyslogd was HUPed Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: logrotate.service: Succeeded. Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: Finished Rotate log files. Jan 06 00:00:06 Eluktronics-Max-17 audit[39592]: AVC apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=39592 comm="cups-browsed" capability=23 capname="sys_nice" Jan 06 00:00:06 Eluktronics-Max-17 kernel: kauditd_printk_skb: 13 callbacks suppressed Jan 06 00:00:06 Eluktronics-Max-17 kernel: audit: type=1400 audit(1641445206.109:25): apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=39592 comm="cups-browsed" capability=23 capname="sys_nice" Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: man-db.service: Succeeded. Jan 06 00:00:06 Eluktronics-Max-17 systemd[1]: Finished Daily man-db regeneration. Jan 06 00:00:07 Eluktronics-Max-17 colord[1173]: failed to get session [pid 39591]: No data available Jan 06 00:00:10 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:00:21 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:01:24 Eluktronics-Max-17 kernel: mce_notify_irq: 4 callbacks suppressed Jan 06 00:01:24 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:01:37 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:02:54 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:02:54 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:03:04 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:04:07 Eluktronics-Max-17 kernel: mce_notify_irq: 2 callbacks suppressed Jan 06 00:04:07 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:04:24 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:05:11 Eluktronics-Max-17 kernel: mce_notify_irq: 2 callbacks suppressed Jan 06 00:05:11 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:05:35 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:06:29 Eluktronics-Max-17 kernel: mce_notify_irq: 1 callbacks suppressed Jan 06 00:06:29 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:06:33 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:07:33 Eluktronics-Max-17 kernel: mce_notify_irq: 4 callbacks suppressed Jan 06 00:07:33 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:07:34 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:08:38 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:08:38 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:08:53 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:09:58 Eluktronics-Max-17 kernel: mce_notify_irq: 3 callbacks suppressed Jan 06 00:09:58 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged Jan 06 00:10:28 Eluktronics-Max-17 kernel: mce: [Hardware Error]: Machine check events logged -- Reboot --