marmot VS payload

If you're willing to accept eventual consistency (a big ask, but acceptable in some scenarios) then there are options like marmot [1] that replicate cdc over nats.

[1]: https://github.com/maxpert/marmot

I've written a whole SQLite replication system that works on top of RAFT ( https://github.com/maxpert/marmot ). Best part is RAFT has a well understood and strong library ecosystem as well. I started of with libraries and when I noticed I am reimplementing distributed streams, I just took off the shelf implementation (https://docs.nats.io/nats-concepts/jetstream) and embedded it in system. I love the simplicity and reasoning that comes with RAFT. However I am playing with epaxos these days (https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf), because then I can truly decentralize the implementation for truly masterless implementation. Right now I've added sharding mechanism on various streams so that in high load cases masters can be distributed across nodes too.

Very interesting! I have question ( out of my experience in https://github.com/maxpert/marmot ) how do get around the boot time, specially when a change log of table is pretty large in Postgres? I've implemented snapshotting mechanism in Marmot as part of quickly getting up to speed. At some level I wonder if we can just feed this PG replication log into NATS cluster and Marmot can just replicate it across the board.

One of the reasons I started writing Marmot (https://maxpert.github.io/marmot/) was for replicating bunch of tables across regions that were read heavy. I even used it for cache replication (because who cares if it’s a cache miss, but a hit will save me time and money). It’s hard to make such blue prints in early days of product, and by the time you hit a true growth almost everyone builds a custom solution for multi-region IAM.

Amazing I was just looking for a good mail server to configure for my demo. Which reminds me since you folks have mentioned LiteStream, have you tried Marmot (https://github.com/maxpert/marmot); I recently configured Isso with Marmot to scale it out horizontally (https://maxpert.github.io/marmot/demo). I am super curious what kind of write workload on a sub thousand people organization will have and if Marmot can help scale it horizontally without Foundation DB. I always find the the convenience of SQLite amazing.

Great that you brought it up. I will fill in the perspective of what I am doing for solving this in Marmot (https://github.com/maxpert/marmot). Today Marmot already records changes via installing triggers to record changes of a table, hence all the offline changes (while Marmot is not running) are never lost. Today when Marmot comes up after a long offline (depending upon max_log_size configuration), it realizes that and tries to catch up changes via restoring a snapshot and then applying rest of logs from NATS (JetStream) change logs. I am working on change that will be publishing those change logs to NATS before it restores snapshots, and once it reapplies those changes after restoring snapshot everyone will have your changes + your DB will be up to date. Now in this case one of the things that bothers people is the fact that if two nodes coming up with conflicting rows the last writer wins.

For that I am also exploring on SQLite-Y-CRDT (https://github.com/maxpert/sqlite-y-crdt) which can help me treat each row as document, and then try to merge them. I personally think CRDT gets harder to reason sometimes, and might not be explainable to an entry level developers. Usually when something is hard to reason and explain, I prefer sticking to simplicity. People IMO will be much more comfortable knowing they can't use auto incrementing IDs for particular tables (because two independent nodes can increment counter to same values) vs here is a magical way to merge that will mess up your data.

Despite being a relatively newer player, Payload's GitHub repository has accumulated 18.8k stars and 1.1K forks as of April 2024, reflecting its growing community. The project has also secured $5.6 million in funding, positioning it for continued growth and innovation.

My most recent project launched in January. NextJS 14 client integrated with PayloadCMS (http://payloadcms.com) for the back-end. I love both technologies in theory, but they're both going through a renaissance period and "bleeding edge" doesn't even begin to describe it.

If I'm just building a client app, create-react-app is still my go to.

Before now, I'd been building on WordPress for 10+ years for anything client-administered. Planning on using Payload from here on out.

Payload CMS: The Customization Insurgent

Payload is free, you can self host it without paying a one time fee or a SaaS fee for its use, it even says so at the bottom of the homepage

James, the co-founder of Payload, a headless CMS with MongoDB support, shared his insights on the drawbacks and limitations of using a headless CMS in the context of web development. He challenged the promises often made about headless CMS, such as separation of concerns and ease of content migration, revealing that these claims often don't align with the reality faced by developers and clients. James is considering integrating Payload directly with Next.js to overcome these limitations and offer a better developer experience, including out-of-the-box features and simpler deployments. Should Payload move to Next.js?

Hey HN, Dan here from Payload (YC S22), an open-source headless CMS that closes the gap between CMS and traditional app frameworks. We’re excited to announce Payload 2.0!

https://github.com/payloadcms/payload

If you’ve not heard of Payload you’re probably wondering why the world needs another CMS. Payload connects to your database and runs without the vendor lock-in and black box of SaaS based CMS solutions, and it’s far more extensible than off-the-shelf SaaS options. Enterprises in specific have been finding value in this control, and they’re using Payload to power content infrastructure that simply isn’t possible through integrating with SaaS webhooks alone.

Today’s announcement is all about features that strike at two neglected areas in the world of CMS. The first is application framework level control over your database that you’d expect with tools like Ruby on Rails or Laravel and the second area is making content editors effective by seeing their edits in realtime.

Here are the highlights on what we’ve been working on:

*Postgres Support*—in the same week we launched about two years ago,people asked for Postgres support. It brings me pure cathartic joy to finally give this to our community. To be fair, MongoDB has been a perfect solution for our architecture and it’s still recommended. But with a new adapter pattern for databases, you can stand your Payload project up on Postgres and run the same functionality as you can with MongoDB now. The crazy part is that we didn’t compromise on how nesting complex fields works. We could have taken the “easy” road and wrote things to JSON, but we leaned fully into the relational way and built the right tables and native column types for fields all the way throughout.

*Database Migrations*—maintaining a production app while deploying schema changes is something you come to expect from ORMs and backend frameworks, but rarely CMS. Payload 2.0 delivers full, first-party migration support all in TypeScript. We took a lot of care on the developer experience here so that when working with Postgres, thanks to our friends at Drizzle, we generate the migration files in TS that add the tables and fields for you. If you have to manipulate data before or after, you have a clear way forward now.

*Database Transactions*—when a request involves multiple inserts, updates or deletes to the database, you need control to rollback all changes when one part fails. The built-in Payload CRUD operations do this now for you and your custom hooks and other code can too.

*Live Preview*—the ability to quickly draft content and see it in context of a website is a literal game changer. We have taken the best dev experience of any headless CMS and given the editors a reason to demand Payload over the others.

*Lexical Richtext Editor*—our original Slate based editor has seen some great features added, like storing related documents directly in the JSON, uploads and any customizations. Unfortunately Slate leaves a lot to be desired on how to extend it, especially compared to Lexical. In a few short weeks we’ve built up a new editor experience inspired by Medium and Notion. Now type “/” and have embedded relationships, uploads, and custom blocks popping right up to be dropped in. Then drag and drop them to reorder your content. If you still want Slate, we continue to support that too.

We’re not compromising on editor experience. This is how we’re bringing the “head” to the headless CMS.

Building critical applications on top of a CMS may sound like blasphemy but it doesn’t have to be that way.

Thanks for reading! I look forward to hearing what you think.