Apache Hadoop VS Seaweed File System

Hadoop (a Big Data tool).

Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data.Wanna dig more dipper?

Few related projects too it on the side of the page here that might be familiar https://hadoop.apache.org/

The computers that you have appear to use an x86 architecture. Therefore, you could most likely install a Linux distro on each one. Then, you could use something like Apache Hadoop to execute some sort of distributed process across each computer.

Hadoop is an ecosystem of tools for big data storage and data analysis. It is older than Spark and writes intermediate results to disk whereas Spark tires to keep data in memory whenever possible, so this is faster in many use cases.

So Yahoo bought that. I think it was 2013 or 2014. Timelines are hard. But I wanted to go join the Games team and start things back up. But that was also my first kind of experience in actually building recommendation engines or working with lots of data. And I think for me, like that was, I guess...at the time, we were using something called Apache Storm. We had Hadoop, which had been around for a while. And it was like one of the biggest user groups was out of the Yahoo campus. It was called the HUG group, like the Hadoop Users Group. So they met for basically pizza and stories on Wednesdays once a month, which was really fun.

Hadoop: http://hadoop.apache.org/

Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.

Apache Hadoop and HDFS

I believe what you want is a horizontally scalable object store with tiered storage. SeaweedFS is free / open source https://github.com/chrislusf/seaweedfs

I don't know if it fits all of your requests, but you can take a look at seaweedfs, which is pretty good

Seaweedfs deserves a mention here for comparison as well.

If you're still talking about SeaweedFS, the answer seems to simply be that it's not a "raft-based object store" as the parent described. That 'proxy' node you mention is a volume server itself, and replicates it's whole volume on another server. Upon replication failures, the data becomes read-only [1]. Raft is not used for the writes.

[1] https://github.com/chrislusf/seaweedfs/wiki/Replication

you could setup something like bcache https://wiki.archlinux.org/title/Bcache on your local system, with your nfs as backend. Updates transfer in the background. That means you wont notice the 'slow' transfer to/from nas. Seaweedfs https://github.com/chrislusf/seaweedfs and similar distributed filesystems also does transfer in the back, so you wont notice it.

For computers, batch IO operations are much faster than random IO and can easily saturate the network.

This benchmark uses large batch size, 64MB, to test. There is nothing new here. Most common file systems can easily do the same.

The difficult task is to read and write lots of small files. There is a term for it, LOSF. I work on SeaweedFS, https://github.com/chrislusf/seaweedfs , which is designed to handle LOSF. And of course, no problem with large files at all.

This is a fair complaint. :)

For filer metadata, you should just pick the one you are most familiar with.

There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup