tangram VS Apache Arrow

Head over to https://www.tangram.dev and give it a try!

In this post, we will go over how we optimized our Gradient Boosted Decision Tree library. This is based on a talk that we gave at RustConf 2021: Writing the Fastest Gradient Boosted Decision Tree Library in Rust. The code is available on GitHub.

You can check out the Tangram Ruby Gem. We built it using Ruby FFI and the source is available on our GitHub repo.

We are using Rust for our web application and website at Tangram. You can view the source here: https://github.com/tangramdotdev/tangram/tree/main/crates/www. The website is at https://www.tangram.dev. We decided to write our own web framework because we needed server rendering and we wanted to use the builder pattern in creating components. Here is an example component for our Logo to get a sense of what this looks like: https://github.com/tangramdotdev/tangram/blob/main/crates/www/ui/logo.rs

Check out https://github.com/tangramdotdev/tangram.

Tangram | Senior Rust Programmer | Remote | https://www.tangram.dev

Tangram is an all in one machine learning framework designed for programmers. With Tangram, developers can train models and make predictions on the command line or with libraries for languages including Elixir, Go, JS, Python, Ruby, and Rust, and learn about their models and monitor them in production from a web application. To learn more about what the product does, watch the demo on the homepage at https://www.tangram.dev or check it out on GitHub at https://www.github.com/tangramdotdev/tangram.

We are looking to grow our engineering team with senior Rust programmers. We are currently based in Boston, MA but are looking to build a remote team. At Tangram, you'll get to work on everything from our core machine learning algorithms to writing front-end code in Rust! We are looking for developers with experience in Rust and familiarity with or willingness to learn machine learning concepts. If this sounds exciting, email me (Isabella, cofounder) at [email protected].

We chose to use Rust instead of TypeScript for the front end of https://github.com/tangramdotdev/tangram.

This allows us to:

* Share code with our server written in Rust.

I'm working on machine learning in Rust at Tangram. We currently only provide an implementation of linear models and gradient boosted decision trees but will move into exposing training of deep models in the future. You can check out Tangram here: https://github.com/tangramdotdev/tangram. You may also be interested in checking out Linfa https://github.com/rust-ml/linfa. If you're interested in the future of machine learning in Rust, check out Luca Palmieri's blog post: https://www.lpalmieri.com/posts/2019-12-01-taking-ml-to-production-with-rust-a-25x-speedup/

In comes Polars: a brand new dataframe library, or how the author Ritchie Vink describes it... a query engine with a dataframe frontend. Polars is built on top of the Arrow memory format and is written in Rust, which is a modern performant and memory-safe systems programming language similar to C/C++.

I learned yesterday about GoLang's assembler https://go.dev/doc/asm - after browsing how arrow is implemented for different languages (my experience is mainly C/C++) - https://github.com/apache/arrow/tree/main/go/arrow/math - there are bunch of .S ("asm" files) and I'm still not able to comprehend how these work exactly (I guess it'll take more reading) - it seems very peculiar.

The last time I've used inlined assembly was back in Turbo/Borland Pascal, then bit in Visual Studio (32-bit), until they got disabled. Then did very little gcc with their more strict specification (while the former you had to know how the ABI worked, the latter too - but it was specced out).

Anyway - I wasn't expecting to find this in "Go" :) But I guess you can always start with .go code then produce assembly (-S) then optimize it, or find/hire someone to do it.

One is related to the heritage of being built around the NumPy library, which is great for processing numerical data, but becomes an issue as soon as the data is anything else. Pandas 2.0 has started to bring in Arrow, but it's not yet the standard (you have to opt-in and according to the developers it's going to stay that way for the foreseeable future). Also, pandas's Arrow-based features are not yet entirely on par with its NumPy-based features. Polars was built around Arrow from the get go. This makes it very powerful when it comes to exchanging data with other languages and reducing the number of in-memory copying operations, thus leading to better performance.

IMO a good first step would be to use the txr FFI to write a library for Apache arrow: https://arrow.apache.org/

https://www.reddit.com/r/O3DE/comments/rdvxhx/why_python/ :

> Python is used for scripting the editor only, not in-game behaviors.

> For implementing entity behaviors the only out of box ways are C++, ScriptCanvas (visual scripting) or Lua. Python is currently not available for implementing game logic.

C++, Lua, and Python all implement CFFI (C Foreign Function Interface) for remote function and method calls.

"Using CFFI for embedding" https://cffi.readthedocs.io/en/latest/embedding.html :

> You can use CFFI to generate C code which exports the API of your choice to any C application that wants to link with this C code. This API, which you define yourself, ends up as the API of a .so/.dll/.dylib library—or you can statically link it within a larger application.

Apache Arrow already supports C, C++, Python, Rust, Go and has C GLib support Lua:

https://github.com/apache/arrow/tree/main/c_glib/example/lua :

> Arrow Lua example: All example codes use LGI to use Arrow GLib based bindings

pyarrow.from_numpy_dtype:

AWS Data Wrangler is a Python library that simplifies the process of interacting with various AWS services, built on top of some useful data tools and open-source projects such as Pandas, Apache Arrow and Boto3. It offers streamlined functions to connect to, retrieve, transform, and load data from AWS services, with a strong focus on Amazon S3.

Worker should really adopt Apache Arrow, which has a much bigger ecosystem.

https://github.com/apache/arrow

Apache Arrow

I am aware of the fact that there are other posts about this issue but none of the ideas to solve it worked for me or sometimes none were found. The issue was discussed in the wheel git hub last December and seems to be solved but then it seems like I'm installing the wrong version? I simply used pip3 install pyarrow, is that wrong?