amazon-eks-ami VS Zabbix

Careful assuming system reserved will be present. Last I checked, AWS EKS does not have system reserved resources for the kubelet by default and as a result, pods can starve those for resources (e.g., https://github.com/awslabs/amazon-eks-ami/issues/79). This is of course more important for memory, but could impact CPU as well.

For example, this is available for AL2: https://github.com/awslabs/amazon-eks-ami

I found this document that lists the pod limits per node size. I suspect you will want to consider larger worker nodes or you will very quickly be unable to schedule additional workloads.

source

Issue 1 is a known issue due to memory reservation being to low, see e.g. https://github.com/awslabs/amazon-eks-ami/issues/1145

No, the new node is added if there are not enough resiurces to start a new pod. So if you have many pods with small resource usage you can hit the pod per node limit, on eks you have a max number of pods depending on the instance type - https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt You can incerase that limit : https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

# of pods are essentially capped by the worker node choice.

below excerpt from: https://github.com/awslabs/amazon-eks-ami/blob/master/files/...

  # Mapping is calculated from AWS EC2 API using the following formula:

See also: EKS nodes lose readiness when containers exhaust memory

So it manifests itself in this way: your pod is scheduled but remains pending forever. You check the logs and you see that it's complaining that the an IP address. Ultimately, if you check here, you see the maximum number of pods that can be scheduled on any underlying ec2 instance, even if you have remaining IPs in your subnet. I found this to be one of the most poorly understood phenomena in EKS. Even those who claimed to "crack" it and wrote fancy blog posts about it fundamentally got it wrong. AFAIK this document reflects the official AWS guide on how to mitigate this.

Zabbix

Zabbix is a comprehensive open-source monitoring solution designed for real-time monitoring and management of various network components, such as servers, networks, and applications. It excels in data collection and processing, enabling proactive monitoring for early issue detection and resolution.

The PHP core package is offered as pre-compiled binary via Ports packages system. In addition, important softwares such as extensions, Composer and PECL libraries are available. So are frameworks such as NextCloud and Zabbix.

Prometheus is a great way to go, however you need to invest time in writing all the alerts. This was a daunting task for us, because we have too many vendors and device types.. we instead went with zabbix, which is a free open source platform similar to Orion. Quite easy to setup. All the device templates for monitoring alerting are provided by the community. There is a grafana plugin which integrates with Zabbix, so you can build beautiful dashboards in grafana while using the polling and alerting logic in zabbix. You can also use grafana OnCall via a zabbix integration. Phase1: You could move to zabbix. Realize cost savings without investing time. Phase2: learn and work on moving things to Prometheus slowly. edit: added links and some rewording

Setup Zabbix (https://www.zabbix.com/) and use SNMP (search for a template for your switches, chances are someone has created one) to pull throughput data from the switchport your proxies are connected to. This will graph them for you on a continual basis, you can then setup some triggers (alerts) that will flag over-utilization (say >80Mbps) for you, can generate an email or SMS alert based on that or just see it in the dashboard.

Look at setting up Zabbix (https://www.zabbix.com/) you can then do ping and latency tests to key hosts/endpoints and get packet loss and other variables in a graphical format. Also will allow you to setup monitoring to alert when there are known issues.

Check out Zabbix. Similar to something like PRTG (I see was already mentioned) but it is a free solution. Only cost is setup time and infrastructure.

now if its more metric data you are using and want to do APM, prometheus is your man https://prometheus.io/, want to make prometheus your full time job? deploy cortex https://cortexmetrics.io/, honorable mention in the metrics space, Zabbix, https://www.zabbix.com/ I've seen use cases of zabbix going way beyond its intended use its a fantastic tool

- NMS / NPM: NetXMS, Zabbix, LibreNMS, PRTG - NCM, updates, automation: Unimus - IaC / automation: Ansible - DCIM / IPAM: NetBox - IPAM / DDI: Infoblox