image-spec VS tcmalloc

A buildpack is a software, designed to transform application source code into executable (OCI) images that can run on a variety of cloud platforms. At its core, a buildpack is a directory that includes a specific file named buildpack.toml. This file contains metadata and configuration details that dictate how the buildpack should behave. Buildpacks in simple terms, is a set of standards defining how the different steps that are required to build a compliant container image can be automated. Using those standards, there are projects that have been built round enabling that using an CLI or an API. The most common way of doing that is through the Cloud Native Buildpacks' Pack project. Pack is a CLI command that can run in the same system the developers are using to actually go through creating a Dockerfile.

Eventually, once zstd support gets fully supported, and tiny gzip compression windows are not a limitation, then compressing a full layer would almost certainly have a better ratio over several smaller layers

https://github.com/opencontainers/image-spec/issues/803

Please note that label-schema has been superseded by https://github.com/opencontainers/image-spec/blob/main/annotations.md<^

GitHub Container Registry stores container images within your organization or personal account, and allows you to associate an image with a repository. It currently supports both the Docker Image Manifest V2, Schema 2 and Open Container Initiative (OCI) specifications.

We build all services as containerized workloads, i.e., OCI images - sometimes called Docker images. We deploy these to the Kubernetes product offered by the cloud vendor. Whenever we need some capability, containers are the answer. This insulates our applications from the vendor. In principle, we could switch providers as long as Kubernetes is available.

Build images with anything that makes OCI compliant images, push, and profit.

it's funny that you mention this because it is actually the thing that is next on my agenda for the image, as you can probably see already I bake in OCI image annotations in our image, which is great for including some core pieces of meta data. In addition to this though I will soon be including custom labels for Base64 encoded YAMLs for Kubernetes deployments using this image. I will look at including helm configuration as well. Then it should be just as easy as: $ docker pull registry.gitlab.com/crafty-controller/crafty-4:latest $ docker image inspect registry.gitlab.com/crafty-controller/crafty-4:latest | jq -r ".[].Config.Labels.\"org.arcadiatech.crafty.k8s.deployment\"" | base64 -d | kubectl apply -f -

They still don't do anything really of substance, they're just gateways to their vendor's world - booking systems, payment systems, etc. You learn those as you go along. Yes, as a potential employee, you need to be able to tick those boxes on your CV, but if you understand the underlying technology, it's mostly a matter of booking your own AWS or Azure server for $5-10 a month for a few weeks, and fooling around. (Docker is a bit different in the sense that they were the first to popularize today's de-facto container image standard, the "Docker container", which has since been accepted as a proper standard and renamed to "OCI image format"; but at the end of the day, at this point in time, Docker in itself is still just a company out for the money, and the multi-GB installation of their product can, for the essential functionality part, be replaced by a few hundred lines of Bash code. The cool boys today don't use Docker, they use [Podman(https://podman.io/), which is essentially a much more lightweight drop-in replacement ;-) )

https://github.com/google/tcmalloc/issues/190

I had earlier raised a query on the github.com/google/tcmalloc regarding how I can force tcmalloc to back memory with hugetlbfs instead of using Transparent Huge Pages. I have attached the link to my query below. Please let me know if there is an possible way to do this.

The allocated_mem_bytes field shows the memory allocated by the memory allocator. PostgreSQL is setup in an extensible way, which includes the ability to choose a memory allocator, which for PostgreSQL is ptmalloc, and for YSQL is tcmalloc. PostgreSQL has the ability to change the memory allocator, but by default uses the operating system memory allocator.

> * Switching from libc malloc to tcmalloc (dating myself a little bit)

If you think of tcmalloc as an old crusty allocator, you've probably only seen the gperftools version of it.

This is the version Google now uses internally: https://github.com/google/tcmalloc

It's worth a fresh look. In particular, it supports per-CPU caches as an alternative to per-thread caches. Those are fantastic if you have a lot more threads than CPUs.

The default setting of max_ptes_none is also problematic.

On a stock kernel, it's 511. TCMalloc's docs recommend using max_ptes_none set to 0 for this reason: https://github.com/google/tcmalloc/blob/master/docs/tuning.m...

(Disclosure: I work on TCMalloc and authored the above doc.)

The easiest way to exploit THP, by far, is to link your program against TCMalloc and forget about it. Literally free money. Highly recommended.

https://github.com/google/tcmalloc

This means that efficient malloc implementation is typically overly complicated. mimalloc for example is almost 8K lines of C afaik, which is one of the smaller but still efficient malloc implementation I'm aware of. (Try looking into tcmalloc for comparison).

Yes, it is synchronized, you can also swap out the implementation typically. There are different allocators out there depending on what you are trying to optimize for (memory, single thread performance, multithread performance, locality, etc). A lot of multithreading optimized ones use per thread pools, so each individual allocation doesn't need to globally lock, but changing the pools themself does, or large allocations that aren't part of the pools. For example https://github.com/google/tcmalloc