meilisearch-js VS Portainer

Thanks! I removed the URLs and now the searchable attributes are only title, description and some author fields!

> Just a detail, if you're making a `du -sh` on your computer, the size on the disk will stay unchanged because we are doing soft deletion ;). Don't worry. It will be physically deleted after a while if you need it in the future.

Ah I was just wildy undershooting the size I gave the PVC! I just gave it much more and it's fine -- right now it's resting around 19Gi of usage, which is actually a bit of a problem considering the data set was only like 4GB or something like that originally. That said, disk is really not an issue so I'll just throw more at it, maybe leave it at 32GB and call it a day (it's around 1.6MM documents out of ~2MM), so shouldn't be too much more.

> If you kept the default configuration of Meilisearch, the maximum size of the HTTP payload is 100Mb (for security). You change it here -> https://docs.meilisearch.com/learn/configuration/instance_op...

Thanks for this, I'll keep this in mind -- so I could actually pass off HUGE chunks to Meilisearch.

It seems like the larger the chunk the more efficient? There didn't seem to be much of a change in how much time it took to work through a chunk of documents, more just that having lots of smaller chunks would go slower. I started off with 10k in a batch, then went to 1k then back to 5k, maybe I should go to 100k docs in a batch and see the performance.

There's a blog post waiting to be written in here...

> addDocumentsInBatches() is just an helper to send your big json array into multiple parts, not absolutely sure you'll need it. (Code -> https://github.com/meilisearch/meilisearch-js/blob/807a6d827...)

Thanks! Was this something someone requested? Is there a tangible benefit (were there some customers that didn't want to split up the payloads themselves)? Because it seems like unnecessary cruft in the API otherwise.

sunspot: looks active, but I would question the idea of using Solr as your search engine. Have you considered https://www.meilisearch.com it a bit like Algolia but open source.

You could use Meilisearch: https://www.meilisearch.com/

COMPANY: Meilisearch, here is our website and Github repository.

Meilisearch - Provides search for the main BookStack static site/docs/blog.

Meilisearch is an open source search engine that adds highly performant search engines to any website or app, including ecommerce stores.

Medusa allows you to integrate any search engine of your choice into the platform. It already integrates with search systems like Meilisearch or Algolia to provide an intuitive search experience for the customers.

I would return a question to you. How much of a database do you need? Wouldn't it be more efficient to implement something like Elastic or https://www.meilisearch.com to index your metadata and quickly return to the users?

Not so long ago I was working on a system based on Keystone.js CMS. But there it was used much more sophisticated way than just as basic headless CMS. I was easily able to extend it with search engine (Rust-based Meilisearch) and connect to external APIs.

Only semi-related, but I've recently started using https://www.meilisearch.com/. It's relatively limited, but works great for small use cases. It's also pretty easy to operate. I'm hoping as it continues to grow it will support more features and use cases. I don't think the creators intend to address the same depth of complex features in ElasticSearch (and the like), but that's a desirable attribute in my opinion.

Portainer

> Any tips on the minimum hardware or VPS's needed to get a small swarm cluster setup?

From my testing, Docker Swarm is very lightweight, uses less memory than both Hashicorp Nomad and lightweight Kubernetes distros (like K3s). Most of the resource requirements will depend on what containers you actually want to run on the nodes.

You might build a cluster from a bunch of Raspberry Pis, some old OptiPlex boxes or laptops, or whatever you have laying around and it's mostly going to be okay. On a practical level, anything with 1-2 CPU cores and 4 GB of RAM will be okay for running any actually useful software, like a web server/reverse proxy, some databases (PostgreSQL/MySQL/MariaDB), as well as either something for a back end or some pre-packaged software, like Nextcloud.

So, even 5$/month VPSes are more than suitable, even from some of the more cheap hosts like Hetzner or Contabo (though the latter has a bad rep for limited/no support).

That said, you might also want to look at something like Portainer for a nice web based UI, for administering the cluster more easily, it really helps with discoverability and also gives you redeploy web hooks, to make CI easier: https://www.portainer.io/ (works for both Docker Swarm as well as Kubernetes, except the Kubernetes ingress control was a little bit clunky with Traefik instead of Nginx)

In this section, we will add Portainer to help us in managing our Docker containers. You can find more details about it here. To integrate Portainer into our EC2 project, we can follow these steps:

Git issues pointing to: https://docs.portainer.io/v/ce-2.9/start/install/agent/swarm/linux gives a ERR_TOO_MANY_REDIRECTS.

I am currently running Portainer, but webhooks (GitOps) appear to be broken ( [2.19.0] GitOps Updates not automatically polling from git · Issue #10309 · portainer/portainer · GitHub ) and so I cannot send webhook to redeploy a stack. So, looking for alternatives. Using this as a good excuse to learn more about docker and CI/CD etc.

A Synology NAS running Portainer (https://www.portainer.io/) running Paperless NGX (https://github.com/paperless-ngx/paperless-ngx)

This works better than I can possibly tell you.

I have an Epson WorkForce ES-580W that I bought when my mother passed away to bulk scan documents and it scans everything, double-sided if required, multi-page PDFs if required, at very high speed and uploads everything to OneDrive, at which point I drag and drop everything into Paperless.

I could, thinking about it, have the scanner email stuff to Paperless. Might investigate that today.

Paperless will OCR it and make it all searchable. This setup is amazing, I love living in the future.

> I've come to the conclusion (after trying kops, kubespray, kubeadm, kubeone, GKE, EKS) that if you're looking for < 100 node cluster, docker swarm should suffice. Easier to setup, maintain and upgrade.

Personally, I'd also consider throwing Portainer in there, which gives you both a nice way to interact with the cluster, as well as things like webhooks: https://www.portainer.io/

With something like Apache, Nginx, Caddy or something else acting as your "ingress" (taking care of TLS, reverse proxy, headers, rate limits, sometimes mTLS etc.) it's a surprisingly simple setup, at least for simple architectures.

casaos it just makes things like backups, offsite syncing and many other nas related things so much easier to manage. And gives you a proper nas like experience similar to that in which you'd fine on companies like tnas or synology. I actually also use it as a replacement for portainer when i don't need the more advanced features it offers

> I moved to docker swarm and love it. It's so much easier, straight forward, automatic ingress network and failover were all working out of the box. I'll stay with swarm for now.

I've had decent luck in the past with the K3s distribution, which is a bit cut down Kubernetes: https://k3s.io/

It also integrates nicely with Portainer (aside from occasional Traefik ingress weirdness sometimes), which I already use for Swarm and would suggest to anyone that wants a nice web based UI: https://www.portainer.io/

Others might also mention K0s, MicroK8s or others - there's lots of options there. But even so, I still run Docker Swarm for most of my private stuff as well and it's a breeze.

For my needs, it has just the right amount of abstractions: stacks with services that use networks and can have some storage in the form of volumes or bind mounts. Configuration in the form of environment variables and/or mounted files (or secrets), some deployment constraints and dependencies sometimes, some health checks and restart policies, as well as resource limits.

If I need a mail server, then I just have a container that binds to the ports (even low port numbers) that I need and configure it. If I need a web server, then I can just run Apache/Nginx/Caddy and use more or less 1:1 configuration files that I'd use when setting up either outside of containers, but with the added benefit of being able to refer to other apps by their service names (or aliases, if they have underscores in the names, which sometimes isn't liked).

At a certain scale, it's dead simple to use - no need for PVs and PVCs, no need for Ingress and Service abstractions, or lots and lots of templating that Helm charts would have (although those are nice in other ways).