pachyderm

Data-Centric Pipelines and Data Versioning (by pachyderm)

Pachyderm Alternatives

Similar projects and alternatives to pachyderm

  • flyte

    pachyderm VS flyte

    Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

  • trivy

    pachyderm VS trivy

    Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • beneath

    Beneath is a serverless real-time data platform ⚡️

  • skaffold

    pachyderm VS skaffold

    Easy and Repeatable Kubernetes Development

  • tsuru

    pachyderm VS tsuru

    Open source and extensible Platform as a Service (PaaS).

  • nebula

    pachyderm VS nebula

    A distributed block-based data storage and compute engine (by varchar-io)

  • dud

    pachyderm VS dud

    A lightweight CLI tool for versioning data alongside source code and building data pipelines.

  • SonarLint

    Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.

  • rexray

    pachyderm VS rexray

    REX-Ray is a container storage orchestration engine enabling persistence for cloud native workloads

  • ESLint

    pachyderm VS ESLint

    Find and fix problems in your JavaScript code.

  • kestra

    pachyderm VS kestra

    Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

  • black

    pachyderm VS black

    The uncompromising Python code formatter

  • ShellCheck

    pachyderm VS ShellCheck

    ShellCheck, a static analysis tool for shell scripts

  • n8n

    pachyderm VS n8n

    Free and source-available fair-code licensed workflow automation tool. Easily automate tasks across different services.

  • dolt

    pachyderm VS dolt

    Dolt – Git for Data

  • dvc

    pachyderm VS dvc

    🦉 Data Version Control | Git for Data & Models | ML Experiments Management

  • Taskflow

    pachyderm VS Taskflow

    A General-purpose Parallel and Heterogeneous Task Programming System

  • openreplay

    pachyderm VS openreplay

    :tv: OpenReplay is developer-friendly, self-hosted session replay.

  • Bup

    pachyderm VS Bup

    Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images). Please post problems or patches to the mailing list for discussion (see the end of the README below).

  • Pylint

    pachyderm VS Pylint

    It's not just a linter that annoys you!

  • orchest

    pachyderm VS orchest

    Build data pipelines, the easy way 🛠️

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better pachyderm alternative or higher similarity.

pachyderm reviews and mentions

Posts with mentions or reviews of pachyderm. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-12-13.
  • Show HN: We scaled Git to support 1 TB repos
    9 projects | news.ycombinator.com | 13 Dec 2022
    There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.

    If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)

    The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.

    Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)

  • Awesome list of VCs investing in commercial open-source startups
    6 projects | /r/opensource | 14 Sep 2022
    Pachyderm - License prevents competition.
  • Airflow's Problem
    6 projects | news.ycombinator.com | 2 Aug 2022
    I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.

    [0] https://github.com/pachyderm/pachyderm

  • One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
    6 projects | dev.to | 7 Mar 2022
    Here is a demo run of lgtm on pachyderm
  • Dud: a tool for versioning data alongside source code, written in Go
    2 projects | /r/golang | 21 Jun 2021
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 4 Jun 2023
    Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression. Learn more →

Stats

Basic pachyderm repo stats
6
5,919
9.7
3 days ago

pachyderm/pachyderm is an open source project licensed under Apache License 2.0 which is an OSI approved license.

The primary programming language of pachyderm is Go.

SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com