k8s-openapi VS datafusion-ballista

It's called sans-io in Python land, which is where I heard it first.

https://sans-io.readthedocs.io/

I did it for one of my projects back in 2018 https://github.com/Arnavion/k8s-openapi/commit/9a4fbb718b119...

Another option is to implement your API in a sans-io form. Since k8s-openapi was mentioned (albeit for a different reason), I'll point out that its API gave you a request value that you could send using whatever sync or async HTTP client you want to use. It also gave you a corresponding function to parse the response, that you would call with the response bytes however you got them from your client.

https://github.com/Arnavion/k8s-openapi/blob/v0.19.0/README....

(Past tense because I removed all the API features from k8s-openapi after that release, for unrelated reasons.)

Macro expansion is slow, but only noticeably in the specific situation of a) third-party proc macros, b) a debug build, and c) a few thousand invocations of said proc macros. This is because debug builds compile proc macros in debug mode too, so while the macro itself compiles quickly (because it's a debug build), it ends up running slowly (because it's a debug build).

I know this from observing this on a mostly auto-generated crate that had a couple of thousand types with `#[derive(serde::)]` on each. [1]

This doesn't affect most users, because first-party macros like `#[derive(Debug)]` etc are not slow because they're part of rustc and are thus optimized regardless of the profile, and even with third-party macros it is unlikely that they have thousands of invocations. Even if it is* a problem, users can opt in to compiling just the proc macros in release mode. [2]

[1]: https://github.com/Arnavion/k8s-openapi/issues/4

[2]: https://github.com/rust-lang/cargo/issues/5622

>OpenAPI Generator allows generation of API client libraries from OpenAPI Specs

It does, but the generated code can be very shitty for some combinations of spec and output language. I maintain Rust bindings for the Kubernetes API server's API, and I chose to write my own code generator instead. The README at https://github.com/Arnavion/k8s-openapi has more details.

k8s_openapi - https://github.com/Arnavion/k8s-openapi

For example: I have a routine that checks the value of (from k8s-openapi): Ingress -> IngressStatus -> LoadBalancerStatus -> Vec[0] -> String

As the maintainer of the Rust bindings that the library used in the article (kube) is backed by, I can confirm that Kubernetes' openapi spec requires a lot of Kubernetes-specific handling to generate a good client than generic openapi generators do not provide.

See https://github.com/Arnavion/k8s-openapi/blob/master/README.m... for a full description.

I also confirm that I keep it up-to-date with Kubernetes releases and have been doing so for the ~3 years that it's been around. Not just the minor ones every few months, but even the point ones; these days the latter usually only involves updating the test cases instead of code changes and they're done within a few hours of the upstream release.

Not super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista

> Is Rust's compile-time GC about something other than performance somehow?

AFAIK, memory safety and language features as RAII is also available in C++, for instance. About the reasons for slow compilation, take a look at https://www.reddit.com/r/rust/comments/xna9mb/why_are_rust_p...

Not having a GC is also about not having a runtime as you mention (e.g. nice for creating Python extensions and embedded systems programming) and also more runtime deterministic performance: on that, if I'm not mistaken that was the reason for Discourse switching to Rust and also, e.g.: "the choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic processing times" https://github.com/apache/arrow-ballista/blob/main/README.md

Ballista (Arrow-Rust), which is largely inspired by Apache Spark, there are some interesting differences.

https://github.com/jorgecarleitao/arrow2 https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-ballista https://github.com/pola-rs/polars https://github.com/duckdb/duckdb

Source: https://github.com/apache/arrow-ballista

Can you explain how this might differ from something like https://github.com/apache/arrow-ballista

I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.

Rust has amazing integration with Python through PyO3 [1] so see it like a safe alternative for high performance calculations. The ecosystem itself is starting to come together exciting projects like Polars [2] (Pandas alternative), nalgebra [3], Datafusion [4] and Ballista [5]

[1] https://github.com/PyO3/pyo3

[2] https://github.com/pola-rs/polars/

[3] https://docs.rs/nalgebra/latest/nalgebra/

[4] https://github.com/apache/arrow-datafusion

[5] https://github.com/apache/arrow-ballista