cosmopolitan VS glibc

> Using a binary for each platform and chip is the past.

Cosmopolitan sends its regards: https://github.com/jart/cosmopolitan

> Rise above with lightweight container[...]

Ah yes, that famously lightweight way of distributing software, full on virtualised containers, each running an OS.

Case in point:

https://github.com/jart/cosmopolitan/blob/master/libc/sysv/s...

The system call numbers of all the unixlikes are bitwise packed into a single number. There is exactly one of those columns which is stable: the Linux one. Everything else is not part of the binary interface of their respective operating systems.

I've written about how Linux is special in this regard:

https://www.matheusmoreira.com/articles/linux-system-calls

It's a neat hack but I'm afraid it's in grave danger of falling victim to the Darth Vader of OS ABI stability.

https://lwn.net/Articles/806870/

> Program to the API rather than the ABI.

> When we see benefits, we change the ABI more often than the API.

> I have altered the ABI.

> Pray I do not alter it further.

Maybe portable CL binaries soon-ish (in CL units of time, mind you)!

Conditional on https://github.com/jart/cosmopolitan/issues/939 as mentioned in INSTALL

Cosmo gives you what you described above and it’s <10kb

https://github.com/jart/cosmopolitan

llamafiles will run on all architectures because they are compiled by cosmopolitan.

https://github.com/jart/cosmopolitan

  Cosmopolitan Libc makes C a build-once run-anywhere language, like Java, except it doesn't need an interpreter or virtual machine. Instead, it reconfigures stock GCC and Clang to output a POSIX-approved polyglot format that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS with the best possible performance and the tiniest footprint imaginable.

My comment was based on cloning master yesterday and trying to build redbean but hitting what looks like https://github.com/jart/cosmopolitan/issues/940

Indeed it lioks like the commit you mentioned should have fixed the issue with the pointer having too many bits for the weird kernel used on android and some raspis. Fingers crossed that release works.

I'll take existence proofs [1] over personal insults but YMMV.

[1] https://github.com/bminor/glibc/commit/7a61e7f557a97ab597d6f...

Depending on how you count, the ratio might not be that small. A lot of hot code are written in hand-coded inline assembly, so in terms of CPU cycles run it's probably non-negligible.

i.e. take a look at the glibc implementation of 'strcmp` [0]

[0] https://github.com/bminor/glibc/blob/master/sysdeps/x86_64/m...

I wonder if you’re using a different definition of ‘vectorized’ from the one I would use. For example glibc provides a vectorized strlen. Here is the sse version: https://github.com/bminor/glibc/blob/master/sysdeps/x86_64/m...

It’s pretty simple to imagine how to write an unoptimized version: read a vector from the start of the string, compare it to 0, convert that to a bitvector, test for equal to zero, then loop or clz and finish.

I would call this vectorized because it operates on 16 bytes (sse) at a time.

There are a few issues:

1. You’re still spending a lot of time in the scalar code checking loop conditions.

2. You’re doing unaligned reads which are slower on old processors

3. You may read across a cache line forcing you to pull a second line into cache even if the string ends before then.

4. You may read across a page boundary which could cause a segfault if the next page is not accessible

So the fixes are to do 64-byte (ie cache line) aligned accesses which also means page-aligned (so you won’t read from a page until you know the string doesn’t end in the previous page). That deals with alignment problems. You read four vector registers at a time but this doesn’t really cost much more if the string is shorter as it all comes from one cache line. Another trick in the linked code is that it first finds the cache line by reading the first 16 bytes then merging in the next 3 groups with unsigned-min, so it only requires one test against a zero vector instead of 4. Then it finds the zero in the cache line. You need to do a bit of work in the first iteration to become aligned. With AVX, you can use mask registers on reads to handle that first step instead.

That was also my thought. To my knowledge `/etc/localtime` is the creation of Arthur David Olson, the founder of the tz database (now maintained by IANA), but his code never read `/etc/localtime` multiple times unless `TZ` environment variable was changed. Tzcode made into glibc but Ulrich Drepper changed it to not cache `/etc/localtime` when `TZ` is unset [1]; I wasn't able to locate the exact rationale, given that the commit was very ancient (1996-12) and no mailing list archive is available for this time period.

[1] https://github.com/bminor/glibc/commit/68dbb3a69e78e24a778c6...

That's not really what the problem is. The actual code is fine.

The issue is that the definition of `fd_set` has a constant size [1]. If you allocate the memory yourself, the select() system call will work with as many file descriptors as you care to pass to it. You can see that both glibc [2] and the kernel [3] support arbitrarily large arrays.

[1] https://github.com/bminor/glibc/blob/master/misc/sys/select....

[2] https://github.com/bminor/glibc/blob/master/sysdeps/unix/sys...

[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

The source code for that is here.

Expanding macro gives three GCC function attributes [2]: `__attribute__ ((malloc))`, `__attribute__ ((alloc_size(1)))` and `__attribute__ ((warn_unused_result))`. They are required for GCC (and others recognizing them) to actually ensure that they behave as the standard dictates. Your own malloc-like functions won't be treated same unless you give similar attributes.

[1] https://github.com/bminor/glibc/blob/807690610916df8aef17cd1...

[2] https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attribute...

IFunc relocations is what enables glibc to dynamically choose the best memcpy routine to use at runtime based on the CPU.

see https://github.com/bminor/glibc/blob/glibc-2.31/sysdeps/x86_...

memmove can be very well implemented in pure C, libc implementations usually have a "generic" (meaning, architecture independent) fallback. Here is musl generic implementation and its x86-64 assembly implementation. For glibc, implementation is a bit more complex, having multiple architectures implemented, but you could find a generic implementation with these two files: memmove.c and generic/memcopy.h.