democratic-csi VS middleware

The interface software regarding is not a worry of mine, as democratic-csi does the storage management for me, thus the compatibility it is not limited to the application using the storage per se, as this is handled by Kubernete's CSI drivers, being application-agnostic when utilizing the storage provided.My main worry is not latency, but rather RAM

Check out https://github.com/democratic-csi/democratic-csi

in k3s, i'm using https://github.com/democratic-csi/democratic-csi (was using iSCSI before, now everything is NFS)

Have you checked this out? GitHub democratic-csi I have yet to test this in my @home K8s cluster. It supports iSCSI volume management for FreeNAS, Synology and other CSI backends.

If you ever migrate from docker up to kubernetes, then take a look into democratic-csi. For a modest homelab, it is a valid option (said from somebody who manages a small homelab and plays around with kubernetes).

For persistent storage have you looked into using TrueNAS with a CSI provider with your container orchestrator? I'm assuming your orchestrator is Nomad or Kubernetes.

Yes, Rancher does support FreeNAS, TrueNAS. and Scale using the storage class provider https://github.com/democratic-csi/democratic-csi It's important to remember Rancher is the server. And in this case, you need to ask the question does the k8s cluster that Rancher is managing support this storage class provider? If you are using RKE the answer is Yes.

I use https://github.com/democratic-csi/democratic-csi to mount nfs/iscsi shares (and manage the shares) from my SAN (truenas box).

iSCSI with democratic-csi (https://github.com/democratic-csi/democratic-csi) works great for me on truenas. I use iSCSI for any PVs that don't need to be shared and NFS for anything I'd like to share between different pods (like movies, music).

Storage - democratic-csi looked the most promising, it has worked well so far. I am using zfs-generic-iscsi against an Ubuntu 20.04 storage server. I also tried zfs-generic-nfs and it worked successfully with the caveat of having to deal with NFS file permissions.

For storage options, most people either purchase a NAS (network attached storage) or re-purpose an older computer using either TrueNAS or unraid. If you're looking to just purchase one, the most popular brand is synology, but their models can be a bit pricey.

https://www.truenas.com/ if you just want to use it as network storage.

Synology if you need prebuilt. If you want to build DIY NAS, take a case that can hold as many drives as you want, take a consumer-grade mobo and Intel/AMD CPU (really doesn't matter for NAS), purchase 1 x SSD for OS and as many drives as you need, deploy something like TrueNAS (https://www.truenas.com/) or Starwinds SAN and NAS (https://www.starwindsoftware.com/san-and-nas), configure RAID (for redundancy, preferably RAID-6) and share the storage to your NUC as iSCSI/NFS/SMB. The second option will require some effort to accomplish but will be more flexible and deliver more performance.

There is no such thing as an ideal OS. Some of the products are better in some of the areas, while other software is better in other areas. For example, Proxmox is the virtualization platform that is targeting virtualization needs. It has support for software RAID, but it doesn't mean that this is the primary feature that is constantly developed. Any NAS OS basically doing the same but targeting storage and sharing things over the virtualization or anything else. So, you need to use whatever is better for the particular use case. Use proxmox on the virtualization host and NAS OS as a storage engine. Or run hypervisor and NAS OS as the VM. As per the alternatives to OMV, you can take a look at Starwinds SAN and NAS (https://www.starwindsoftware.com/san-and-nas), TrueNAS (https://www.truenas.com/), or even pure Debian + Cockpit (https://cockpit-project.org/)

The second is storage. If you need any of the storage sharing, deploy NAS OS as the VM in proxmox, like Starwinds SAN and NAS (https://www.starwindsoftware.com/san-and-nas) or OMV (https://www.openmediavault.org/), or TrueNAS (https://www.truenas.com/). As you mentioned, you need to cross-flash the perc into IT mode and pass through the controller into VM, but you need a separate from the controller drive for proxmox to be able to PCI-E passthrough the card into VM. Then, configure software RAID and reshare the storage to the proxmox via NFS/iSCSI (that will improve your skills in storage stack and storage protocols).

You can also go another route with Proxmox and NAS OS as a VM. TrueNAS or Starwinds SAN&NAS can be used. https://www.truenas.com/

You would need some sort of NAS device to act as a file server (you obviously can't just plug a HDD directly into a switch). Some consumer routers have USB ports where you can plug in an external HDD, though they frequently have speed issues with the USB ports. You could buy something from QNAP, Synology, etc. or build your own TrueNAS.

I run plex on my lab, but if I didn't have that, I would probably buy a cheap server and run TrueNAS. https://www.truenas.com/ I personally have a whole vmware network using it for storage, but as just a plex server with a bunch of storage would be a viable alternative for a single host and have the storage plus the raid benefits of not having to worry about disk failures.

Take a look at TrueNAS or Starwinds SAN&NAS as a NAS OS options. https://www.truenas.com/

Windows built-in iSCSI server is slow. If you don't mind replacing it, try using TrueNAS, Starwinds SAN&NAS, or even Linux (Ubuntu Server) VM running the iSCSI target server. Either solution should overperform the Windows alternative.