vector VS engine

job "vector" { datacenters = ["dc1"] # system job, runs on all nodes type = "system" group "vector" { count = 1 network { port "api" { to = 8686 } } ephemeral_disk { size = 500 sticky = true } task "vector" { driver = "docker" config { image = "timberio/vector:0.30.0-debian" ports = ["api"] volumes = ["/var/run/docker.sock:/var/run/docker.sock"] } env { VECTOR_CONFIG = "local/vector.toml" VECTOR_REQUIRE_HEALTHY = "false" } resources { cpu = 100 # 100 MHz memory = 100 # 100MB } # template with Vector's configuration template { destination = "local/vector.toml" change_mode = "signal" change_signal = "SIGHUP" # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }} left_delimiter = "[[" right_delimiter = "]]" data=<

we are doing something similar with OTEL but we are looking at using https://vector.dev/

We don't pull logs, we forward logs to a centralized logging service.

opensearch - amazon fork of Elasticsearch https://opensearch.org/docs/latestif you do this an have distributed log sources you'd use logstash for, bin off logstash and use vector (https://vector.dev/) its better out of the box for SaaS stuff.

I have done something similar in the past: you can send the logs through a centralized syslog servers (I suggest syslog-ng) and from there ingest into ELK. For parsing I am advice to use something like Vector, is a lot more faster than logstash. When you have your logs ingested correctly, you can create your own dashboard in Kibana. If this fit your requirements, no need to install nginx (unless you want to use as reverse proxy for Kibana), php and mysql.

I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.

Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.

A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.

- [1] https://vector.dev

Log to stdout/stderr and collect your logs with a tool like vector (vector.dev) and send it to something like Grafana Loki.

I would recommend that you run vector as a systems service so you don't have to worry about managing it. Here is a basic config to do that - https://github.com/vectordotdev/vector/blob/master/distribution/systemd/vector.service .

Qovery: Qovery simplifies the deployment process for Kubernetes by letting you declare your project's structure and dependencies, making it an excellent choice for startups looking to focus on development without worrying about infrastructure.

Enter Qovery - an Internal Developer Platform that can be installed on any Kubernetes cluster to bring a delightful and simplified developer experience. But don't take my word for granted. In this quick step-by-step article, I will show you how to deploy one of my GitHub apps and a Postgres database on my Kubernetes cluster running on GCP. And what's really really cool is that I don't even need to understand how Kubernetes work!

Qovery is a comprehensive PaaS tool that offers a range of features to simplify the deployment and management of applications. With Qovery, you can easily deploy your applications in various cloud environments, eliminating the complexities of infrastructure setup. Its intuitive and developer-friendly interface, Git integration, and automatic scaling capabilities make it a powerful choice for building a custom PaaS. Qovery's seamless integration with popular frameworks and databases further enhances its appeal.

Qovery https://www.qovery.com

I have one of my customers who want to use Qovery to deploy their apps on AWS China. Technically speaking, Qovery runs on Kubernetes and on all the usual AWS regions. But AWS China (Beijing) is a bit different (cf here)

Combining GitLab (CI) with Qovery (CD) sounds what you need.

I am eager to get your feedback on Qovery - an open-source CD platform that helps thousands of developers and DevOps to build and scale on AWS.

Qovery - an open-source platform that makes AWS easier and there is a Terraform provider available