arc VS anarki

A while ago I posted a link to [Arc](https://news.ycombinator.com/item?id=26573930) a declarative method for defining repeatable data pipelines which execute against [Apache Spark](https://spark.apache.org/).

Today I would like to present a proof-of-concept implementation of the [Arc declarative ETL framework](https://arc.tripl.ai) against [Apache Datafusion](https://arrow.apache.org/datafusion/) which is an Ansi SQL (Postgres) execution engine based upon Apache Arrow and built with Rust.

The idea of providing a declarative 'configuration' language for defining data pipelines was planned from the beginning of the Arc project to allow changing execution engines without having to rewrite the base business logic (the part that is valuable to your business). Instead, by defining an abstraction layer, we can change the execution engine and run the same logic with different execution characteristics.

The benefit of the DataFusion over Apache Spark is a significant increase in speed and reduction in execution resource requirements. Even through a Docker-for-Mac inefficiency layer the same job completes in ~4 seconds with DataFusion vs ~24 seconds with Apache Spark (including JVM startup time). Without Docker-for-Mac layer end-to-end execution times of 0.5 second for the same example job (TPC-H) is possible. * the aim is not to start a benchmarking flamewar but to provide some indicative data *.

The purpose of this post is to gather feedback from the community whether you would use a tool like this, what features would be required for you to use it (MVP) or whether you would be interested in contributing to the project. I would also like to highlight the excellent work being done by the DataFusion/Arrow (and Apache) community for providing such amazing tools to us all as open source projects.

Disclosure: I am a contributor to Datafusion.

I have done a lot of work in the ETL space in Apache Spark to build Arc (https://arc.tripl.ai/) and have ported a lot of the basic functionality of Arc to Datafusion as a proof-of-concept. The appeal to me of the Apache Spark and Datafusion engines is the ability to a) seperate compute and storage b) express transformation logic in SQL.

Performance: From those early experiments Datafusion would frequently finish processing an entire job _before_ the SparkContext could be started - even on a local Spark instance. Obviously this is at smaller data sizes but in my experience a lot of ETL is about repeatable processes not necessarily huge datasets.

Compatibility: Those experiments were done a few months ago and the SQL compatibility of the Datafusion engine has improved extremely rapidly (WINDOW functions were recently added). There is still some missing SQL functionality (for example to run all the TPC-H queries https://github.com/apache/arrow-datafusion/tree/master/bench...) but it is moving quickly.

It still is a single single-core server, dang references it frequently when there's unusually high traffic [0]. And the language you're referring to is Arc [1]. They do have caching for not-logged-in users, historically done through nginx [2]. From other comments in this thread, it sounds like they just temporarily put Cloudflare in front of that single server to block a DDoS.

[0] https://news.ycombinator.com/item?id=38310213

[1] https://arclanguage.github.io/

[2] https://news.ycombinator.com/item?id=26473226

Slashdot, reddit, and HN are similar in that the source code was available. For HN, as part of arc under the Artistic license. All 3 abandoned public source code releases.

https://sourceforge.net/projects/slashcode/

https://github.com/arclanguage/anarki

https://github.com/reddit-archive/reddit

>Where is the HN source code right now? Free and Open?

Yes and no.

HN itself is running a proprietary fork of Arc Lisp, which you can find here[0]. The Arc maintainters don't take public PRs or feature requests, and HN itself has numerous changes to the codebase which aren't public for business reasons.

There is a public fork of Arc called Anarki[1] which has no direct connection to HN or Arc Lisp, and for which the community and development is... well... anarchic.

And given the general culture here around minimalism and stasis (not wanting to introduce new features for fear of entropy that would negatively affect the signal to noise ratio and push the site towards Eternal September) chances are it's not likely to happen.

But dang's email is at the bottom of the page if you want to ask him.

[0]http://arclanguage.org/

[1]https://github.com/arclanguage/anarki

The original version was open sourced (Perl artistic License) http://arclanguage.org/ There is an active fork in https://github.com/arclanguage/anarki but it's totally independent and the current conde in HN can be (very) different.

My guess is that it's very difficult to keep all the details of the secret sauce hidden. They change the details very often. For example the front page is ordered by points/time^1.6, but the 1.6 changes from time to time without notice (I think it was 1.8 for some time, perhaps it's 1.8 or something else now. Some people have analyzed the front page and got compatible results, but I don't remember the exponent they found and I'm too lazy to try).

You might find something useful here:

https://github.com/arclanguage/anarki/tree/master/apps/news

It was published in the public version of 2009 https://github.com/arclanguage/anarki/blob/9f2e1dd53b5b66bb4... Look for "contro-factor" (i.e. controversial-factor).

This is old code, and the mods make tweaks here and there without warning, so the details may have changed. (I'm not sure if gravity changed from 1.8 to 1.6 (???).)

I didn't test it personally, but it feels like HN is using something very similar. And there are a few black box analysis of the sorting of the front page that got similar results. Also, minimaxir is the kind of person that is probably running an script to use the HN API to verify the claims.

Absolutely. It's basically the "community version" of Paul Graham's Arc.

Not to mention arc/anarki (hn is/was written in arc - anarki comes with a "news" example app/forum):

https://github.com/arclanguage/anarki/tree/HEAD#readme

arc [1] / racket implimentation of demystifycpp might be able to provide something 'usable' at the command line / straight up web browser html file.

[1] : https://arclanguage.github.io/