pachyderm VS dark

20. Pachyderm | Github | tutorial

Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.

There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.

If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)

The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.

Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)

Pachyderm - License prevents competition.

I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.

[0] https://github.com/pachyderm/pachyderm

Here is a demo run of lgtm on pachyderm

Yes. Darklang was originally in OCaml using js_of_ocaml, and we ported it to F# using Blazor (https://github.com/darklang/dark/tree/main/backend/src/Wasm). It works.

We found that in dotnet 6, the code was much slower, with long startup times and a much bigger download, than in js_of_ocaml. It also had a lot of issues in running in a Webworker, which wasn't the case for js_of_ocaml.

In dotnet 7, the webworker issues are better and AOT is easier, so startup is faster. Download sizes are still bad, and it's still slower than js_of_ocaml.

However, dotnet allows almost any code to run in WASM, which js_of_ocaml had large limitations. This meant a decent chunk of functionality had to be worked around to make separate js vs native targets, which also was a massive pain and took a long time. Dune's virtual targets wasn't ready at the time - I think we were one of the test cases for it.

Why should I use this instead of https://darklang.com/

First it was probably Dark. They made a lot of noise some years ago, but then I never heard of them again (looking at their current website, looks like they moved on to AI now, obviously).

Reminds me of Darklang. Personally, I don't think vendoring cloud services into a language is going to be beneficial. I'm curious how the language deals with vendor updates. Do I have to upgrade the language then? If so, I see a lot conflicts coming from this. Then it comes down to Javascript or HCL, the HCL bit makes me think that the below statement is not as truthy as it is on the surface:

We still don't have all that many users (~100 active), so I'm not sure you'll find an answer here. But we collect that sort of feedback publicly, which might answer your question: https://github.com/darklang/dark/discussions/categories/feed...

I wish I could remember what took me to YCombinator's website on the 10th of October, 2022. That was when I first heard about Wasp and another language called DarkLang. After I learned about Wasp, I was intrigued and curious to know how it works, which led me to join the discord server the next day.

Some languages that try to integrate an HTTP server and a database:

Ur/Web: http://impredicative.com/ur/

Dark (Darklang): https://darklang.com/

There are others in this space I did not assess like Encore, Shuttle, Modal, and Dark. These were not assessed for the sake of time. If you're interested in IfC, I encourage you to take a look at these others.

Oh, finally! I was waiting to build my serverless CRUD webapp in Dark (OCaml + JavaScript and Fsharp?) until they had support for returning negative numbers on a GET request!