GlusterFS VS NATS

https://github.com/gluster/glusterfs/issues/894

And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.

This Product Appendix does not apply to online service offerings managed by Red Hat or generally available open source projects such as www.wildfly.org, www.fedoraproject.org, www.openstack.redhat.com, www.gluster.org, www.centos.org, okd.io, Ansible Project Software or other community projects.

Just because Red Hat will stop selling commercial support for their product, does not mean GlusterFS itself is dying. It's an open source project like any other - https://github.com/gluster/glusterfs

https://www.gluster.org/ Is the way to do this across nodes

This allows us to fetch the desired quality of the video as per the user's request, and once the media file finishes processing, it will be uploaded to a distributed file storage such as HDFS, GlusterFS, or an object storage such as Amazon S3 for later retrieval during streaming.

GlusterFS

But where can we store files at scale? Well, object storage is what we're looking for. Object stores break data files up into pieces called objects. It then stores those objects in a single repository, which can be spread out across multiple networked systems. We can also use distributed file storage such as HDFS or GlusterFS.

GlusterFS - same as above (https://www.gluster.org/)

What you look for is a clustered file system. Like https://www.gluster.org/. As long as all units are closeby with low latency there are a couple solutions that allow you to create distributed storage solutions of various kinds. Key value stores applenty, clustered file systems that pretent to be one file system etc. If you have geographically distributed solutions with high latencies it becomes harder. Most open source systems don't work really well in this scenario. There were a couple attempts like Hydrabase but they didn't go so far. It normally is solved by doing two clusters and then replicate between them.

I'd imagine most managed service providers are going to require a credit card, though most of them have a free tier. If you want to take an unmanaged approach, maybe look into Gluster. I've used it before and never had issue with it, but I also had an infrastructure team that set it up, so I'm not familiar with the challenges that way: https://www.gluster.org/

Several message brokers, such as NATS and database queues, are not supported by OpenTelemetry (OTel) SDKs. This article will guide you on how to use context propagation explicitly with these message queues.

https://nats.io/ (Tracker removed)

> Connective Technology for Adaptive Edge & Distributed Systems

> An Introduction to NATS - The first screencast

I guess I don't need to know what it is

During our interview, we referred to NATS quite a few times! If you want to learn more about it, Sebastian suggests this tutorial series.

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.

[1] https://nats.io/

Imagine you have an AI-powered personal alerting chat assistant that interacts using up-to-date data. Whether it's a big move in the stock market that affects your investments, any significant change on your shared SharePoint documents, or discounts on Amazon you were waiting for, the application is designed to keep you informed and alert you about any significant changes based on the criteria you set in advance using your natural language. In this post, we will learn how to build a full-stack event-driven weather alert chat application in Python using pretty cool tools: Streamlit, NATS, and OpenAI. The app can collect real-time weather information, understand your criteria for alerts using AI, and deliver these alerts to the user interface.

Why wasn't NATS[1] used ?

Written in Go, single-binary deployment... there's a lot to love about NATS !

[1]https://nats.io/

require nats.io

About cost, see [1]. Also, S3 prices have been increasing and there's been a bunch of alternative offers for object store from other companies. I think people in here (HN) comment often about increasing costs of AWS offerings.

Distributed systems and consensus are inherently hard problem, but there are a lot of implementations that you can study (like Etcd that you mention, or NATS [2], which I've been playing with and looks super cool so far :-p) if you want to understand the internals, on top of many books and papers released.

Again, I never said it was "easy" to build distributed systems, I just don't think there's any esoteric knowledge to what S3 provides.

--

1: https://en.wikipedia.org/wiki/Economies_of_scale

2: https://nats.io/

I am in a project were nats.io is used. Someone thought, it would be a great idea to link data in an event with data in a response using the response channel name.