linux-hardened VS systemd

Glossing over their hardening guide, we notice that the kernel-hardened package is mentioned. This is a fork of what once was the kernel of GrapheneOS. While this hardened kernel can be used on a variety of distros, unfortunately this doesn't apply to Fedora Silverblue. Furthermore, I haven't seen any mention of the hardened kernel being used on openSUSE Tumbleweed. Therefore I see no reason to believe that this is possible on openSUSE Aeon either. Though, I'd love to be corrected on this!

Here's the GitHub page for the linux-hardened package in the official Arch repos. You will find that lots of changes come from GrapheneOS. "Forked from GrapheneOS" is in tiny, tiny text near the top-left of the web page underneath the project name.

Here is the source code for the hardened Linux kernel.

so that we can use it liberally in CLI) > **Additionally, here's a 'building from the source section** - https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md#building-from-source ### Running Firecracker > "*In production, Firecracker is designed to be run securely, inside an execution jail, carefully set up by the jailer binary. This is how our integration test suite does it. However, if you just want to see Firecracker booting up a guest Linux machine, you can do that as well.*" 1. We need to first obtain an "uncompressed Linux kernel binary, and an ext4 file system image (to use as rootfs)" ; great, these are two things that we need to seek out before we move forward in our 'adventure' (*this really feels like a "quest" of some sort, like the ones that they forced you to play on Runescape back in the days*) **How to Decompress Linux Kernel** (explicit instructions to be honest here) - https://0xax.gitbooks.io/linux-insides/content/Booting/linux-bootstrap-5.html **Linux-Hardened Kernel** - https://github.com/anthraxx/linux-hardened (this is something that they're all still actively working on at this very point in time) They also say that we need an 'ext4 file system image' (where do we obtain this from?) - found it **Full Guide on How to Create an EXT4 filesystem image here** -https://fabianlee.org/2020/01/13/linux-mounting-a-loopback-ext4-xfs-filesystem-to-isolate-or-enforce-storage-limits/ Assuming that the above has been handled, the directions insist that we create two separate shell prompts, (one to run Firecracker, and another one to control it [by writing to the API socket]; both shells have to run "in the same directory where the firecracker binary was placed") ^^ What? - This is a pain in the ass because this is something that they should've mentioned earlier (obv. everyone is going to move a binary where the rest of their binaries go ; and you're not going to just load up some random project to be used in that manner) - Not even sure what the end goal of opening up an API socket here would really be But fuck it, let's just assume that we play ball and we adhere to all of these (additional) steps that we're being put through (just for the setup up this virtualization tool!). ### Following Through on the Next Steps 1. Ensuring that Firecracker can create its own API ``` bash rm -f /tmp/firecracker.socket

Officially supported kernels Community support on forum and bug reporting is available for officially supported kernels. Stable — Vanilla Linux kernel and modules, with a few patches applied. https://www.kernel.org/ || linux Hardened — A security-focused Linux kernel applying a set of hardening patches to mitigate kernel and userspace exploits. It also enables more upstream kernel hardening features than linux. https://github.com/anthraxx/linux-hardened || linux-hardened Longterm — Long-term support (LTS) Linux kernel and modules. https://www.kernel.org/ || linux-lts Zen Kernel — Result of a collaborative effort of kernel hackers to provide the best Linux kernel possible for everyday systems. Some more details can be found on https://liquorix.net (which provides kernel binaries based on Zen for Debian). https://github.com/zen-kernel/zen-kernel || linux-zen

Overall those patches do protect you in a lot of cases. Per default upstream features are chose to honor security first. Furthermore patches do frequently protect against upstream failures and vulnerabilities. One of the latest examples is CVE-2022-1729 https://seclists.org/oss-sec/2022/q2/122 a local privilege vulnerability that linux-hardened simply protects since before day 0: https://github.com/anthraxx/linux-hardened/commit/4dd6bdf3b079ef73e597661ee961d225bfccbe2a On top the approach in several places of fail-early instead of potentially continuing with known corruption or certain use-after-free regularly uncovers problems that get fixed upstream and where the vanilla kernels just keep on running, which include exploitable problems. One example of a faulty very recent commit that has been bisected and fixed because of linux-hardened: https://git.kernel.org/pub/scm/linux/kernel/git/deller/linux-fbdev.git/commit/?h=for-next&id=acde4003efc16480375543638484d8f13f2e99a3

Those two don't protect the kernel itself from "misuse". To do that, there are ways like kernel hardening 3, using hardened kernel 4 5.

https://github.com/systemd/systemd/issues/32028#issuecomment...

There are some very compelling arguments made there if you care to read them

Maybe it was because you weren't pointing out anything new?

There was a pull request to stop linking libzma to systemd before the attack even took place

https://github.com/systemd/systemd/pull/31550

This was likely one of many things that pushed the attackers to work faster, and forced them into making mistakes.

The PR for changing compression libraries to use dlopen() was opened several weeks before the xz-utils backdoor was revealed.

https://github.com/systemd/systemd/pull/31550

I find this the most plausible explanation by far:

* The highly professional outfit simply did not see teknoraver's commit to remove liblzma as standard dependency of systemd build scripts coming.

* The race was on between their compromised code and that commit. They had to win it, with as large a window as possible.

* This caused serious errors.

* The performance regression is __not__ big. It's lucky Andres caught it at all. It's also not necessarily all that simple to remove it. It's not simply a bug in a loop or some such.

* The payload of the 'hack' contains fairly easy ways for the xz hackers to update the payload. They actually used it to remove a real issue where their hackery causes issues with valgrind that might lead to discovering it, and they also used it to release 5.6.1 which rewrites significant chunks; I've as yet not read, nor know of any analysis, as to why they changed so much.

Extra info for those who don't know:

https://github.com/systemd/systemd/commit/3fc72d54132151c131...

That's a commit that changes how liblzma is a dependency of systemd. Not because the author of this commit knew anything was wrong with it. But, pretty much entirely by accident (although removing deps was part of the point of that commit), almost entirely eliminates the value of all those 2 years of hard work.

And that was with the finish line in sight for the xz hackers: On 24 feb 2024, the xz hackers release liblzma 5.6.0 which is the first fully operational compromised version. __12 days later systemd merges a commit that means it won't work__.

So now the race is on. Can they get 5.6.0 integrated into stable releases of major OSes _before_ teknoraver's commit that removes liblzma's status as direct dep of systemd?

I find it plausible that they knew about teknoraver's commit _just before_ Feb 24th 2024 (when liblzma v5.6.0 was released, the first backdoored release), and rushed to release ASAP, before doing the testing you describe. Buoyed by their efforts to add ways to update the payload which they indeed used - March 8th (after teknoraver's commit was accepted) it was used to fix the valgrind issue.

So, no, I don't find this weird, and I don't think the amateurish aspects should be taken as some sort of indication that parts of the outfit were amateuristic. As long as it's plausible that the amateuristic aspects were simply due to time pressure, it sounds like a really bad idea to make assumptions in this regard.

sshd is started by systemd.

systemd has several ways of starting programs and waiting until they're "ready" before starting other programs that depend on them: Type=oneshot, simple, exec, forking, dbus, notify, ...

A while back, several distro maintainers found problems with using Type=exec (?) and chose Type=notify instead. When sshd is ready, it notifies systemd. How you do notification is you send a datagram to systemd's unix domain socket. That's about 10 lines of C code. But to make life even simpler, systemd also provides the one-line sd_notify() call, which is in libsystemd.so. This library is so other programmers can easily integrate with systemd.

So the distro maintainers patched sshd to use the sd_notify() function from libsystemd.so

What else is in libsystemd.so? That's right, systemd also does logging. All the logging functions are in there, so user programs can do logging the systemd way. You can even _read_ logs, using the functions in libsystemd.so. For example, sd_journal_open_files().

By the way... systemd supports the environment variable SYSTEMD_JOURNAL_COMPRESS which can be LZ4, XZ or ZSTD, to allow systemd log files to be compressed.

So, if you're a client program, that needs to read systemd logs, you'll call sd_journal_open_files() in libsystemd.so, which may then need liblz4, liblzma or libzstd functions.

These compression libraries could be dynamically loaded, should sd_journal_open_files() need them - which is what https://github.com/systemd/systemd/pull/31550 submitted on the 29th February this year did. But clearly that's not in common use. No, right now, most libsystemd.so libraries have headers saying "you'll need to load liblz4.so, liblzma.so and libzstd before you can load me!", so liblzma.so gets loaded for the logging functions that sshd doesn't use, so the distro maintainers of sshd can add 1 line instead of 10 to notify systemd that sshd is ready.

They just added an example to the documentation[0] of how to implement the sd_notify protocol without linking to libsystemd, so a little bit of discarding systemd (or at least parts of it) does seem to be part of the solution.

[0] https://github.com/systemd/systemd/pull/32030/files

I think this analysis is more interesting if you consider these two events in particular:

2024-02-29: On GitHub, @teknoraver sends pull request to stop linking liblzma into libsystemd.[1]

2024-03-20: The attacker is now a co-contributor for a patchset proposed to the Linux kernel, with the patchset adding the attacker as a maintainer and mirroring activity with xz-utils.

A theory is that the attacker saw the sshd/libsystemd/xz-utils vector as closing soon with libsystemd removing its dependency on xz-utils. When building a Linux kernel image, the resulting image is compressed by default with gzip [3], but can also be optionally compressed using xz-utils (amongst other compression utilities). There's a lot of distributions of Linux which have chosen xz-utils as the method used to compress kernel images, particularly embedded Linux distributions.[4] xz-utils is even the recommended mode of compression if a small kernel build image is desired.[5] If the attacker can execute code during the process of building a new kernel image, they can cause even more catastrophic impacts than targeting sshd. Targeting sshd was always going to be limited due to targets not exposing sshd over accessible networks, or implementing passive optical taps and real time behavioural analysis, or receiving real time alerts from servers indicative of unusual activity or data transfers. Targeting the Linux kernel would have far worse consequences possible, particularly if the attacker was targeting embedded systems (such as military transport vehicles [6]) where the chance of detection is reduced due to lack of eyeballs looking over it.

[1] https://github.com/systemd/systemd/pull/31550

[2] https://lkml.org/lkml/2024/3/20/1004

[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

[4] https://github.com/search?q=CONFIG_KERNEL_XZ%3Dy&type=code

[5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

[6] https://linuxdevices.org/large-military-truck-runs-embedded-...

systemd merged a change to using dlopen for compression libraries recently https://github.com/systemd/systemd/pull/31550 which is a safer linking method in that sense.

1) Debian includes this downstream patch, also.

2) A potential explanation for "why now" is that systemd DID prevent these dependencies from loading automatically in a patch one month ago, and the patches enabling the backdoor merged a few days later. It could be a total coincidence or it could be that the attacker was trying to catch the window before it was closed on them https://github.com/systemd/systemd/pull/31550#issuecomment-1...