wg-meshconf
seaweedfs
Our great sponsors
wg-meshconf | seaweedfs | |
---|---|---|
6 | 34 | |
877 | 21,013 | |
- | 2.3% | |
0.0 | 9.9 | |
15 days ago | 6 days ago | |
Python | Go | |
GNU General Public License v3.0 only | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
wg-meshconf
- Wireguard mesh between 4 pc similar to Tailscale
-
Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put
my experience, i dont know if this is comparable, but from my memory (i have not made any notes on that), i've tried min.io in december and switched to seaweed a weeks ago, because my usecase was transition from local file storage to DFS + also enable our developers to transition from local filesystem to s3. Since my resources are limited (vsphere VM) with 3 hosts + different disks, i tried to set up a 3 vm cluster with minio first, after i did some research on different systems (ceph, longhorn.io, ..) i wanted to have an easy setup-able system, which supports s3. I relied a lot on what people measured and chose min.io first because it supported mount via s3. Then i tried to copy over about 34 million files (mostly few bytes, but can also be 1Gbyte), with a mass of about 4.2TB. I tried different methods, rsync, cp, cp with parallelism,.. and i took me about 3 days to copy over 300GB of data at best. Then i also found out that it was impossible to list files. We have one single folder with over 300k projects (guid) beneath (growing). After that i gave seaweed a shot. Why i did not used it firsthand was documentation was a bit confusing and it did not gave me all the answers i needed as fast as minio did.
Now, my seaweed setup is a 3 vm cluster with 3 disks per vm (1TB) each. I configured a wireguard mesh (https://github.com/k4yt3x/wg-meshconf) between the VMs and configured master and volumes server to talk to each other via wireguard IPs securely. I also configured ufw to only allow communication between http/gRPC ports. I also configured a filer (using leveldb3) to use wireguard IPs (master and volumes) and let it communicate with some specific servers on the outside (ufw).
After that i mounted the filer via weed.mount on that specific server and tried to copy over the same files/folders. after 2 days i copied over about 1.5 TB of the data via rsync. There was also no problem with file listing and accessing the filer from different machines while uploading stuff. But there is a overhead when reading and creating lots of small files. File listing is even faster than local btrfs file listing.
chris is also very nice and fast fixing bugs.
-
Connect to wireguard server over a wireguard server -> client connection
Hey you should post your wg0.conf If you would like to build a WireGuard mesh try this: https://github.com/k4yt3x/wg-meshconf
-
How to add new client to wireguard in VPS without getting public IP changed on the client?
There are two factors at play here. The client's public IP actually depends on the gateway they use on accessing the internet. You can disable routing and your clients will keep their public IP and general internet access won't go through the VPS. However, if you want the traffic between "clients" also skip the VPS, then you want a mesh network. wesher and wg-meshconf can help you on configuring them.
-
Wiretrustee: WireGuard-Based Mesh Network
Looks great!
I've been using wg-meshconf[1] to assist in setting up Wireguard Mesh Networks on Linux for a while, works amazing!
A massive use case is to setup Kubernetes clusters, where network encryption is extremely important.
[1]: https://github.com/k4yt3x/wg-meshconf
- WireGuard full mesh configuration generator
seaweedfs
-
DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System
Whoops: WebDAV:
https://news.ycombinator.com/item?id=39417503
SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV
I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.
It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.
-
Billion File Filesystem
If you want/need to take out the metadata, there's some nice solutions for that https://github.com/seaweedfs/seaweedfs
-
SeaweedFS fast distributed storage system for blobs, objects, files and datalake
I posted this on https://github.com/seaweedfs/seaweedfs/discussions/5290
-
DuckDB + dbt for a serverless event correlation pipeline?
I like the idea of using SeaweedFS as an intermediate layer with object write notifications going to SQS, RabbitMQ, or a local file, which could also allow me to observe the changes to different files through a metric collection layer like Prometheus and Grafana.
-
Show HN: OpenSign – The open source alternative to DocuSign
> Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.
Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing
I've actually been looking for more open alternatives, but haven't found much.
Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)
There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/
The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs
- The Tailscale Universal Docker Mod
- SeaweedFS
- Google Cloud Storage FUSE
- Experience running rook-ceph in production/large clusters
-
First Homelab as a 19yr old Software Developer
SeaweedFS S3 Gateway for Joplin notes
What are some alternatives?
wesher - wireguard overlay mesh network manager
minio - The Object Store for AI Data Infrastructure
headscale - An open source, self-hosted implementation of the Tailscale control server
Ceph - Ceph is a distributed object, block, and file storage platform
tinc - a VPN daemon
garage - (Mirror) S3-compatible object store for small self-hosted geo-distributed deployments. Main repo: https://git.deuxfleurs.fr/Deuxfleurs/garage
cjdns - An encrypted IPv6 network using public-key cryptography for address allocation and a distributed hash table for routing.
cubefs - cloud-native file store
netbird - Connect your devices into a single secure private WireGuard®-based mesh network with SSO/MFA and simple access controls.
GlusterFS - Web Content for gluster.org -- Deprecated as of September 2017
Netmaker - Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
MooseFS - MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)