-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I'm aware that encapsulated control flow like this exists at a syntactic level, but the implementation deficiencies remain unaddressed:
1. No access to the actual concrete type of the Stream, as you mentioned, which means you can't do Vec - you have to resort to indirections like Vec>>. You might be able to work around this somehow if you were determined enough (DWARF debug info and std::mem::transmute come to mind), but any such approach would result in an ugly and exceedingly brittle abomination.
2. Brittle, if any, optimization. `stream!` is basically async/await with proc macro syntax dressing, and Rust's async/await implementation still suffers the same sort of bloated frame size issues as generators do, under certain circumstances.
To provide some context, suppose I want to write a custom event loop that handles millions of concurrent sessions (could be filesystem transactions, or TCP sessions, or some other I/O). At that level, every kilobyte of per-session data I add equates to additional gigabytes of memory usage. Every byte of space used by the compiler-generated state machine has to be carefully accounted for. I can't afford to have my per-connection context blowing up in size simply because the compiler naively duplicates stack slots for arguments across yield points [1], or because one of my local variable types implements Copy [2], or because the compiler will simply fail to optimize local variables that are never live across a yield point [3]. I suppose if I care so much about memory usage, I could just draw up my own state machine, mentally track the liveness of each state variable and lay out a hand-written size-optimized struct accordingly, but that's going to be a painful exercise for reasons stated in the original article.
[1] https://github.com/rust-lang/rust/issues/62958
What's really unfortunate is that compilers already perform this "control flow -> state machine" transformation, but almost none of them expose it to the user for direct manipulation - instead, they tightly couple it to a bunch of unrelated abstractions like event loop runtimes, async executors, managed stacks, etc. I'd kill for a compiler that can properly:
* Parse a coroutine that produces and consumes values
* Perform all the normal function-level optimizations to minimize frame space (stack slot reuse via liveness analysis of local variables, re-computing temporaries across yield points, etc.)
* Expose that coroutine as a fixed-size struct that I can explicitly resume and query
Zig is almost there, but suspend/resume cannot return values/take arguments, which requires some unergonomic workarounds. Rust has unified coroutines (https://github.com/rust-lang/rfcs/pull/2781), but the generator types are opaque so you can't encapsulate them in a struct or allocate an array of them. Not to mention that it's still extra-unstable, and last I checked, there were issues with generator size optimizations (https://github.com/rust-lang/rust/issues/59087). C++20 coroutines are similarly opaque and cannot be allocated as a contiguous array.