|Apache Hadoop||Seaweed File System|
|about 22 hours ago||7 days ago|
|GNU General Public License v3.0 or later||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Python vs. Java: Comparing the Pros, Cons, and Use Cases
3 projects | dev.to | 21 May 2022
Hadoop (a Big Data tool).
Pokemon vs Programming
1 project | reddit.com/r/WorkReform | 12 Apr 2022
Big Data Processing, EMR with Spark and Hadoop | Python, PySpark
2 projects | dev.to | 27 Mar 2022
Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data.Wanna dig more dipper?
Unknown Python.exe process taking 2% CPU
1 project | reddit.com/r/WindowsHelp | 2 Feb 2022
Few related projects too it on the side of the page here that might be familiar https://hadoop.apache.org/
How do I make multiple computers run as one?
1 project | reddit.com/r/techsupport | 6 Jan 2022
The computers that you have appear to use an x86 architecture. Therefore, you could most likely install a Linux distro on each one. Then, you could use something like Apache Hadoop to execute some sort of distributed process across each computer.
Spark for beginners - and you
3 projects | dev.to | 22 Dec 2021
Hadoop is an ecosystem of tools for big data storage and data analysis. It is older than Spark and writes intermediate results to disk whereas Spark tires to keep data in memory whenever possible, so this is faster in many use cases.
Dreaming and Breaking Molds – Establishing Best Practices with Scott Haines
3 projects | dev.to | 8 Dec 2021
So Yahoo bought that. I think it was 2013 or 2014. Timelines are hard. But I wanted to go join the Games team and start things back up. But that was also my first kind of experience in actually building recommendation engines or working with lots of data. And I think for me, like that was, I guess...at the time, we were using something called Apache Storm. We had Hadoop, which had been around for a while. And it was like one of the biggest user groups was out of the Yahoo campus. It was called the HUG group, like the Hadoop Users Group. So they met for basically pizza and stories on Wednesdays once a month, which was really fun.
Setting up a single-node Hadoop cluster
1 project | dev.to | 14 Nov 2021
Spark is lit once again
6 projects | dev.to | 29 Oct 2021
Here at Exacaster Spark applications have been used extensively for years. We started using them on our Hadoop clusters with YARN as an application manager. However, with our recent product, we started moving towards a Cloud-based solution and decided to use Kubernetes for our infrastructure needs.
The Data Engineer Roadmap 🗺
11 projects | dev.to | 19 Oct 2021
Apache Hadoop and HDFS
Seaweed File System
SeaweedFS and YDB
1 project | news.ycombinator.com | 8 May 2022
Cost effective managed key-value store?
1 project | reddit.com/r/Database | 28 Apr 2022
I believe what you want is a horizontally scalable object store with tiered storage. SeaweedFS is free / open source https://github.com/chrislusf/seaweedfs
A way to store and query large (up to 1GB) user defined objects.
1 project | reddit.com/r/Database | 13 Mar 2022
Question: does anyone know Storage Provider with S3 as persistence layer?
2 projects | reddit.com/r/kubernetes | 15 Feb 2022
I don't know if it fits all of your requests, but you can take a look at seaweedfs, which is pretty good
Introducing Garage, our self-hosted distributed object storage solution
2 projects | reddit.com/r/selfhosted | 8 Feb 2022
Seaweedfs deserves a mention here for comparison as well.
Garage, our self-hosted distributed object storage solution
10 projects | news.ycombinator.com | 8 Feb 2022
If you're still talking about SeaweedFS, the answer seems to simply be that it's not a "raft-based object store" as the parent described. That 'proxy' node you mention is a volume server itself, and replicates it's whole volume on another server. Upon replication failures, the data becomes read-only . Raft is not used for the writes.10 projects | news.ycombinator.com | 8 Feb 2022
Tuning server for fast writes?
1 project | reddit.com/r/selfhosted | 7 Feb 2022
you could setup something like bcache https://wiki.archlinux.org/title/Bcache on your local system, with your nfs as backend. Updates transfer in the background. That means you wont notice the 'slow' transfer to/from nas. Seaweedfs https://github.com/chrislusf/seaweedfs and similar distributed filesystems also does transfer in the back, so you wont notice it.
Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put
3 projects | news.ycombinator.com | 4 Jan 2022
For computers, batch IO operations are much faster than random IO and can easily saturate the network.
This benchmark uses large batch size, 64MB, to test. There is nothing new here. Most common file systems can easily do the same.
The difficult task is to read and write lots of small files. There is a term for it, LOSF. I work on SeaweedFS, https://github.com/chrislusf/seaweedfs , which is designed to handle LOSF. And of course, no problem with large files at all.3 projects | news.ycombinator.com | 4 Jan 2022
This is a fair complaint. :)
For filer metadata, you should just pick the one you are most familiar with.
There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup
What are some alternatives?
Ceph - Ceph is a distributed object, block, and file storage platform
minio - Multi-Cloud Object Storage
GlusterFS - Web Content for gluster.org -- Deprecated as of September 2017
Go IPFS - IPFS implementation in Go
MooseFS - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)
lizardfs - LizardFS is an Open Source Distributed File System licensed under GPLv3.
autotier - A passthrough FUSE filesystem that intelligently moves files between storage tiers based on frequency of use, file age, and tier fullness.