Ignite – Use Firecracker VMs with Docker-Like UX

• ignite

  20 3,473 0.0 Go

  Discontinued Ignite a Firecracker microVM (by weaveworks)

  

> If it's not hard to name a thing that Firecracker makes difficult for a serverside workload, could you... name one?

I already did that with live migration. But ok.

Encrypted storage: https://github.com/firecracker-microvm/firecracker/issues/65...: WONTFIX

The answer given is appropriate for firecracker use cases but insufficient otherwise. I'm not anti-firecracker; it's the right choice for many things. Just not all things.

The sort of VM I want orchestrated has encrypted (by contract) multi-pathed network block devices to encrypted storage volumes. 3-10 per tenant. This is trivial for a full-featured kernel; multi-path just works, encryption just works.

VLAN: https://github.com/weaveworks/ignite/issues/810: Open. Maybe one day.

Again, trivial for a full-featured Linux kernel.

I think you're missing the point. It's not about what hypothetical thing firecracker can or can't do. It's about elevating VM orchestration to some degree of parity with what has been created for container orchestration. These VMs and their complex storage and networking requirements should be modeled as we model containers now; through an orchestration system that makes management easy and as foolproof as possible. The fact that firecracker isn't sufficient to be the Micro-VM of choice for this isn't relevant.

• firecracker

  75 24,084 9.9 Rust

  Secure and fast microVMs for serverless computing.

  

I don't have direct experience for examples, but just to hypothesize:

- Firecracker/KVM can introduce some CPU overhead, for which there are some workarounds (https://github.com/firecracker-microvm/firecracker/blob/main...)

- Firecracker networking uses a tap device. Want any kind of advanced networking hardware, say, to offload packet processing, do network inspection, etc, and access it from the containerized app? Not gonna work... Have some advanced network controller that transparently handles mesh networks, or some other fancy shmancy system designed to manage complex interactions between containers and networks? Probably not gonna work due to assumptions about what lies between the layers, what components use what tricks to handle advanced routing (Netfilter, eBGP, sidecars, etc). To say nothing of link-level differences (what if your network isn't Ethernet?). And all traffic is copies from an I/O thread of an emulated network device to a host TAP device; my guess is network latency and maximum PPS were not a consideration.

- The exposed CPU is based on what KVM supports exposing to guests; presumably QEMU supports a much wider selection. Clocksources, of which I know absolutely nothing (:D), are only allowed as kvm-clock.

- All I/O is rate-limited by a custom scheme built by the Firecracker authors. I'm sure this is fine for most use-cases, but some weird high-performance outlier is gonna hit some sort of bottleneck with this thing I'm sure.

- Firecracker emulated block devices are backed by files on the host. Ergo, any app that wants to control a disk directly, or use some fancy shmancy SAN directly, etc is out of luck.

- The guest requires a balloon driver to use balloon support, which means the guest needs this special software, and compromising it could be a serious issue. I don't know if Kata does this differently.

- Aarch64 support has a bunch of errata currently, so x86_64 is the only fully-supported platform; I dunno if Kata does any better, but this is a real hardware limitation, esp. if you're trying to buy a shit-ton of cheap powerful machines.

- This is not hardware-specific, but Firecracker seems limited to specific kernel releases; right now the latest it supports is 5.10, according to https://github.com/firecracker-microvm/firecracker/blob/main...

I would add that containers are used in far more settings than server-side, and would be great to have on the edge, in IoT, and on desktops, if they were a little less... funky. In general, containers requires a lot of extra steps to be "usable" as general purpose applications, and access to hardware will absolutely be a barrier we need to cross in order for "general purpose containerized apps" to become commonplace.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project