Apache Hadoop VS GlusterFS

Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data.Wanna dig more dipper?

Few related projects too it on the side of the page here that might be familiar https://hadoop.apache.org/

The computers that you have appear to use an x86 architecture. Therefore, you could most likely install a Linux distro on each one. Then, you could use something like Apache Hadoop to execute some sort of distributed process across each computer.

Hadoop is an ecosystem of tools for big data storage and data analysis. It is older than Spark and writes intermediate results to disk whereas Spark tires to keep data in memory whenever possible, so this is faster in many use cases.

So Yahoo bought that. I think it was 2013 or 2014. Timelines are hard. But I wanted to go join the Games team and start things back up. But that was also my first kind of experience in actually building recommendation engines or working with lots of data. And I think for me, like that was, I guess...at the time, we were using something called Apache Storm. We had Hadoop, which had been around for a while. And it was like one of the biggest user groups was out of the Yahoo campus. It was called the HUG group, like the Hadoop Users Group. So they met for basically pizza and stories on Wednesdays once a month, which was really fun.

Hadoop: http://hadoop.apache.org/

Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

Apache Hadoop and HDFS

What you look for is a clustered file system. Like https://www.gluster.org/. As long as all units are closeby with low latency there are a couple solutions that allow you to create distributed storage solutions of various kinds. Key value stores applenty, clustered file systems that pretent to be one file system etc. If you have geographically distributed solutions with high latencies it becomes harder. Most open source systems don't work really well in this scenario. There were a couple attempts like Hydrabase but they didn't go so far. It normally is solved by doing two clusters and then replicate between them.

I'd imagine most managed service providers are going to require a credit card, though most of them have a free tier. If you want to take an unmanaged approach, maybe look into Gluster. I've used it before and never had issue with it, but I also had an infrastructure team that set it up, so I'm not familiar with the challenges that way: https://www.gluster.org/

Gluster is developed in Github at https://github.com/gluster/glusterfs/

I'm running three Blocky instances in Docker (and CoreDNS for internal zone resolving) by placing YAML files on a GlusterFS share, so I can update configs on one VM, and then just restart Blocky containers via SSH.

I've you've got a few similar computers available (Like Pis, though not sure how well it runs on Arm) you could set up a little Gluster and show off how node failure and recovery affect a distributed file system.

Longhorn and storage in general the hardest part of any HA setup, but also not the only choice, at the most basic level something like glusterFS is easy to get running and usable in k8s as NFS volumes, it however doesn't have all the extra features of longhorn.

GlusterFS (distributed storage)

No, this package isn't available in Solus Repository. If you are going to compile it, the source code is here: https://github.com/gluster/glusterfs

Do you mean https://www.gluster.org/ ?