pachyderm

Data-Centric Pipelines and Data Versioning (by pachyderm)

Pachyderm Alternatives

Similar projects and alternatives to pachyderm

  1. ShellCheck

    499 pachyderm VS ShellCheck

    ShellCheck, a static analysis tool for shell scripts

  2. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  3. Pandas

    424 pachyderm VS Pandas

    Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

  4. black

    336 pachyderm VS black

    The uncompromising Python code formatter

  5. n8n

    329 pachyderm VS n8n

    Fair-code workflow automation platform with native AI capabilities. Combine visual building with custom code, self-host or cloud, 400+ integrations.

  6. OpenCV

    Open Source Computer Vision Library

  7. dvc

    121 pachyderm VS dvc

    🦉 Data Versioning and ML Experiments

  8. dolt

    97 pachyderm VS dolt

    Dolt – Git for Data

  9. InfluxDB

    InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.

    InfluxDB logo
  10. MLflow

    74 pachyderm VS MLflow

    Open source platform for the machine learning lifecycle

  11. label-studio

    50 pachyderm VS label-studio

    Label Studio is a multi-type data labeling and annotation tool with standardized output format

  12. Ray

    47 pachyderm VS Ray

    Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

  13. orchest

    44 pachyderm VS orchest

    Build data pipelines, the easy way 🛠️

  14. dark

    Darklang main repo, including language, backend, and infra

  15. Flake8

    38 pachyderm VS Flake8

    flake8 is a python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code.

  16. Pylint

    29 pachyderm VS Pylint

    It's not just a linter that annoys you!

  17. Taskflow

    A General-purpose Task-parallel Programming System using Modern C++

  18. Bup

    20 pachyderm VS Bup

    Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).

  19. awesome-production-machine-learning

    A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

  20. scikit-image

    Image processing in Python

  21. determined

    Determined is an open-source machine learning platform that simplifies distributed training, hyperparameter tuning, experiment tracking, and resource management. Works with PyTorch and TensorFlow.

  22. beneath

    Discontinued Beneath is a serverless real-time data platform ⚡️

  23. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better pachyderm alternative or higher similarity.

pachyderm discussion

Log in or Post with

pachyderm reviews and mentions

Posts with mentions or reviews of pachyderm. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-01-05.
  • Open Source Advent Fun Wraps Up!
    10 projects | dev.to | 5 Jan 2024
    20. Pachyderm | Github | tutorial
  • Exploring Open-Source Alternatives to Landing AI for Robust MLOps
    18 projects | dev.to | 13 Dec 2023
    Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.
  • Show HN: We scaled Git to support 1 TB repos
    9 projects | news.ycombinator.com | 13 Dec 2022
    There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.

    If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)

    The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.

    Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)

  • pachyderm: Data-Centric Pipelines and Data Versioning
    1 project | /r/u_TsukiZombina | 5 Dec 2022
  • Awesome list of VCs investing in commercial open-source startups
    6 projects | /r/opensource | 14 Sep 2022
    Pachyderm - License prevents competition.
  • Airflow's Problem
    6 projects | news.ycombinator.com | 2 Aug 2022
    I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.

    [0] https://github.com/pachyderm/pachyderm

  • One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
    6 projects | dev.to | 7 Mar 2022
    Here is a demo run of lgtm on pachyderm
  • Dud: a tool for versioning data alongside source code, written in Go
    2 projects | /r/golang | 21 Jun 2021
  • A note from our sponsor - InfluxDB
    influxdata.com | 25 Apr 2025
    Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems. Learn more →

Stats

Basic pachyderm repo stats
8
6,222
9.5
3 months ago

pachyderm/pachyderm is an open source project licensed under Apache License 2.0 which is an OSI approved license.

The primary programming language of pachyderm is Go.


Sponsored
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai

Did you know that Go is
the 4th most popular programming language
based on number of references?