fastglobal VS kafka_ex

> State has to be owned by a process - who owns the state of the world? ... Keep in mind, when data is passed "between" processes it is deep copied.

The Erlang runtime has other stateful things besides processes, and manipulating data within these and/or getting a handle on data in these places doesn't necessarily involve copying.

• ETS tables, which hold state privately to themselves (with copying in/out) in a similar way to how processes do. If you manipulate data inside ETS tables by "sending compute to data" (think Redis INCR) rather than by "sending data to compute", then no copying happens.

• "Globals" in the form of data compiled into read-only versioned modules loaded into the modules table and available to be referenced from any/all active processes, only copied if the module gets unloaded before the process dies. (Originally this was just a design pattern — https://github.com/discord/fastglobal — but it eventually became its own runtime feature in Erlang 22, https://www.erlang.org/doc/man/persistent_term).

• Large binaries (anything over 64 bytes) aren't allocated in an actor's memory arena, but instead are allocated in a special global-per-node binaries heap, and then ref-counted, where each actor-process holds one reference to each large-binary it's using, and then each read-only slice of that binary, in turn holds a reference to the per-actor reference handle for the binary. One clever technique for sharing a large "database" of data between many actors, is to store the data encoded in a large binary in an encoding that is efficient to partwise-decode; and have the "lookup" operation just parse+decode the appropriate data out of the binary. (This is how erlang:module_info/2 used to work — the global modules-table itself holding a set of references to the loaded modules' binaries, which module_info would then parse metadata out of on demand.)

I am roughly 3 weeks into working with Elixir and so far it has been pretty good! I currently am working in a system that has a couple of Golang services running as producers and consumers within a Kafka cluster. I've been working with the Kafka_ex library and have been able to get messages and create topics. It was decided that messages being sent through Kafka would be in Protobuf format, so I've started to use this protobuf library to handle converting into structs.