pachyderm
Taskflow
Our great sponsors
pachyderm | Taskflow | |
---|---|---|
8 | 24 | |
6,071 | 9,520 | |
0.3% | 1.7% | |
9.8 | 7.9 | |
8 days ago | 8 days ago | |
Go | C++ | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pachyderm
-
Open Source Advent Fun Wraps Up!
20. Pachyderm | Github | tutorial
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.
-
Show HN: We scaled Git to support 1 TB repos
There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
- pachyderm: Data-Centric Pipelines and Data Versioning
-
Awesome list of VCs investing in commercial open-source startups
Pachyderm - License prevents competition.
-
Airflow's Problem
I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.
[0] https://github.com/pachyderm/pachyderm
-
One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
Here is a demo run of lgtm on pachyderm
- Dud: a tool for versioning data alongside source code, written in Go
Taskflow
-
Improvements of Clojure in his time
For parallel programming nowadays, personally I reach for C++ Taskflow when I really care about performance, or a mix of core.async and running multiple load balanced instances when I’m doing more traditional web backend stuff in Clojure.
- Taskflow: A General-Purpose Parallel and Heterogeneous Task Programming System
-
How to go from intermediate to advance in C++?
Also, you can take a look to good libraries. The problem is that very often libraries are heavily templated, so It could be hard. For example, I like the style of the Taskflow library, I think is very clear, is relatively small, while makes use of more advanced techniques: https://github.com/taskflow/taskflow
-
gcl v1.1 released - Graph Concurrent Library for C++
Cool. Thanks! How does it compare to taskflow?
-
std::execution from the metal up - Paul Bendixen - Meeting C++ 2022
I've not seen yet, but it's been a bit since I looked last, any evidence of being able to build a computation graph and "save" it to re-run on new inputs. Something like https://github.com/taskflow/taskflow
-
Proper abstraction for this?
It seems you're describing something a generic parallel task framework. Check taskflow for a production ready example https://github.com/taskflow/taskflow/blob/master/
-
That one technology, question, or skill you never learned, and now you are haunted by during every new job conversation...
- https://github.com/taskflow/taskflow (I recommend to learn it first since its API and documentation are excellent)
-
Parallel Computations in C++: Where Do I Begin?
If you want some sort of "job" system, where you submit items to a some sort of queue to be processed in parallel, try searching for a thread pool - there isn't one in the standard library, but there's about a million implementations online. There are more complicated versions of that idea, that describe computation as a directed acyclic graph, such as taskflow.
-
High level overview of my custom game engine
The tooling decisions affect engine design though. For example if you want to have visual representation of job graph as it happened in specific frame of interest you need to pass the information around about job relationships and output it to a tool of choice. For example see https://github.com/taskflow/taskflow
-
Is there any good reason not to build an open-source C++ project on Intels oneTBB?
I am aware of DAGs of task based threading library like Taskflow and HPX however the benefit they have is not obvious to me, as the following sequential section depends on the parallel part being completed fully. If you want to suggest elaboration on the benefits of this approach would be welcome.
What are some alternatives?
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
tbb - oneAPI Threading Building Blocks (oneTBB) [Moved to: https://github.com/oneapi-src/oneTBB]
trivy - Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
tensorflow - An Open Source Machine Learning Framework for Everyone
dud - A lightweight CLI tool for versioning data alongside source code and building data pipelines.
HPX - The C++ Standard Library for Parallelism and Concurrency
beneath - Beneath is a serverless real-time data platform ⚡️
C++ Actor Framework - An Open Source Implementation of the Actor Model in C++
typhoon-orchestrator - Create elegant data pipelines and deploy to AWS Lambda or Airflow
entt - Gaming meets modern C++ - a fast and reliable entity component system (ECS) and much more
tsuru - Open source and extensible Platform as a Service (PaaS).
libunifex - Unified Executors