We Run Data Pipelines as Containers

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • enhancements

    Enhancements tracking repo for Kubernetes

  • Make sure you are at least using user namespaces which drop the mknod cap by their nature, or better yet in rootless mode.

    I have filed several bugs that I know can result in breakout but as I can't make myself disclose vulnerabilities I have no stick to get them to change their 'wont fix' decisions.

    k8s doesn't support user namespaces let alone user mount namespaces.

    Here is the open issue.

    https://github.com/kubernetes/enhancements/issues/127

    The point being that for k8s and docker, any role that allows you to create pods/containers or to compromise such a role with any hop number should be considered as having root permissions.

    While I won't share any non-privileged breakouts, here is an example of how easy it is with the --privileged flag.

    https://stackoverflow.com/questions/36425230/privileged-cont...

    While I am not recommending it in general, AppArmor is fairly easy to develop CI friendly restrictions with and I would strongly suggest you protect the directory space and devices that you don't use with it.

    Not perfect but it typically can help prevent leaks caused by adding features or configuration errors.

    Runc using seccomp to make a container process make the one-way transition into a "secure" state and through dropped capabilities is what provides additional security.

    Hiding pids doesn't matter when any container can list /dev or look through /sys and /proc to find device major and minor numbers or to modify kernel parameters or files that are mistakenly given write access.

    The overwriting of the runc executable CVE that recently happened will give an actual case there.

    Namespaces are more about decoupling and avoiding pollution than security.

    Just like chroot, the shared kernel instance has a large attack surface, especially if you don't leverage all of the tools provided.

    As you are effectively running arbitrary code from users, I would highly suggest you look into non container runtime protection.

    It can be made reasonably safe but an overconfidence in containers being inherently secure will make you a target.

    If you are on k8s you should be using anti-affinity or taints to make sure containers running external user code is not running on the same nodes as other containers or better than that have a dedicated k8s for that need.

    Especially if you have persistent storage as user mount point namespaces are new in the kernel and default mounts typically are implemented by granting CAP_SYS_ADMIN capabilities(7)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Design Docs at Google

    3 projects | news.ycombinator.com | 7 May 2024
  • IBM to buy HashiCorp in $6.4B deal

    1 project | news.ycombinator.com | 25 Apr 2024
  • Isint release cycle becoming a bit crazy with monthly releases and deprecations ?

    2 projects | /r/kubernetes | 11 Jul 2023
  • What's there in K8s 1.27

    1 project | dev.to | 4 Jun 2023
  • Dependencies between Services

    1 project | /r/kubernetes | 6 Apr 2023