perspective VS Apache Arrow

We're looking for senior product managers and engineers of all experience levels to build the next generation of collaborative data visualization. At the Prospective Co., you'll contribute to our existing open-source project as well as help design our enterprise offering.

https://perspective.finos.org/

We're looking for any of:

- Familiarity with WebAssembly, data visualization, WebGL/OpenGL, data science, Jupyter/notebook, web/desktop/mobile UI development, compiler/language or database design, finance services.

- Primary stack is Rust (targeting WebAssembly). JavaScript, C++ and Python are a big plus.

- We <3 GitHub contributors - opt to discuss your GitHub work in lieu of a technical interview.

Contact [email protected]

DESCRIPTION: We're looking for senior product managers and engineers of all experience levels to build the next generation of collaborative data visualization. At the Prospective Co., you'll contribute to our existing open-source project (Perspective https://perspective.finos.org/) as well as help design our enterprise offering. We're looking for any of: - Familiarity with WebAssembly, data visualization, WebGL/OpenGL, data science, Jupyter/notebook, web/desktop/mobile UI development, compiler/language or database design, finance services. - Primary stack is Rust (targeting WebAssembly, especially Yew). JavaScript, C++ and Python are a big plus. - We <3 GitHub contributors - opt to discuss your GitHub work in lieu of a technical interview.

Thank you. I saw this, https://github.com/finos/perspective

Several members of the Ruby community have been deeply involved in the development of Apache Arrow.

In that case, why not use polars, which supports apache arrow format which supports C, C++, Rust, Python and supports zero-copy read.

I think the naming will likely cause some confusion with apache arrow. My initial thoughts when reading "Introducing ArrowJS" was a new port of the apache arrow spec.

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.

Just another embedded SQL engine.

There are SQLite(OLTP), DuckDB(OLAP) and some engine-based project like mentioned Apache Arrow(https://arrow.apache.org/)(OLAP): Apache Arrow has many language implementations, some do not include the query engine(for example, Rust implementation, which depends on the DataFusion for more SQL-like analytics) in its own repo, but other do include(for example, C++).

There is a comprehensive benchmark by ClickHouse for OLAP but including kinds of embedding engines: https://benchmark.clickhouse.com/

The more interesting is that, in fact, we have not an embedded HTAP engine. One of my database products already implements 3/4 HTAP at the engine layer, but unfortunately it's still just a free software, not an open source implementation.

Apache Arrow

This is a meta-request for the library, but imo it would be really awesome if it used a data structure compatible with Arrow: https://arrow.apache.org/

As a bit of an aside, you could imagine a way to get the best of both worlds with an extension to Docker that would allow you to publish a container that exposes a Python API, so that someone could call sentiment = call_container_api(image="huggingface/transformers", "my input text") directly from their python code. This would effectively be a remote procedure call into a container that is not running as a service but instead spun up just for the purpose of executing a function on-demand. This feels like a really heavyweight approach to solving dependency hell, but if your libraries are using a cross-platform memory format (hello Apache Arrow!) under the covers, you could imagine doing some fun tricks like giving the container a read-only view into the caller’s memory space to reduce the overhead. It’s a bit implausible, but sometimes it’s helpful to sketch out these ideas to clarify the tradeoffs we’re making with the more practical bits of technology we have available.

I assume you mean to use Apache arrow rather than scala Arrow?