Bizarre memory leak caused by tokio runtime

• jemallocator

  2 302 5.2 Rust

  Rust allocator using jemalloc as a backend (by tikv)

  

I was able to replicate the issue with the sample code, but not when using a different allocator (https://github.com/tikv/jemallocator). I understand the default allocator on Linux is malloc (https://doc.rust-lang.org/std/alloc/struct.System.html), and while looking up the differences between the two, it seems like jemalloc handles memory fragmentation much better (https://engineering.linkedin.com/blog/2021/taming-memory-fragmentation-in-venice-with-jemalloc). Maybe that's what's going on?

• tokio

  196 24,677 9.5 Rust

  A runtime for writing reliable asynchronous applications with Rust. Provides I/O, networking, scheduling, timers, ...

  

This link specifically has more info on why the memory could creep up when you spawn a lot of tasks.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Mesh

  6 1,702 3.7 C++

  A memory allocator that automatically reduces the memory footprint of C/C++ applications.

  

With everyone talking about memory fragmentation, I'd like to mention Mesh, an allocator that can compact aka defrag the heap without any help from the program or compiler. Here's the talk explaining it, "Compacting the Uncompactable" by Bobby Powers.

• rkyv

  13 2,555 8.6 Rust

  Zero-copy deserialization framework for Rust

  

I had the same problem when trying to deserialize a big struct with rkyv: see rkyv#277.

Suggest a related project