pachyderm
skaffold
Our great sponsors
pachyderm | skaffold | |
---|---|---|
8 | 83 | |
6,074 | 14,659 | |
0.3% | 0.8% | |
9.8 | 9.2 | |
2 days ago | 7 days ago | |
Go | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pachyderm
-
Open Source Advent Fun Wraps Up!
20. Pachyderm | Github | tutorial
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.
-
Show HN: We scaled Git to support 1 TB repos
There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
- pachyderm: Data-Centric Pipelines and Data Versioning
-
Awesome list of VCs investing in commercial open-source startups
Pachyderm - License prevents competition.
-
Airflow's Problem
I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.
[0] https://github.com/pachyderm/pachyderm
-
One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
Here is a demo run of lgtm on pachyderm
- Dud: a tool for versioning data alongside source code, written in Go
skaffold
- Google to Discontinue Skaffold
-
You've just inherited a legacy C++ codebase, now what?
A nice middle ground is using a tool like Google's Skaffold, which provides "Bazel-like" capabilities for composing Docker images and tagging them based on a number of strategies, including file manifests. In my case, I also use build args to explicitly set versions of external dependencies.
While I am in a Typescript environment with this setup at the moment, my personal experience that Skaffold with Docker has a lighter implementation and maintenance overhead than Bazel. (You also get the added benefit of easy deployment and automatic rebuilds.)
I quite liked using Bazel in a small Golang monorepo, but I ran into pain when trying to do things like include third-party pre-compiled binaries in the Docker builds, because of the unusual build rules convention. The advantage of Skaffold is it provides a thin build/tag/deploy/verify layer over Docker and other container types. Might be worth a look!
Kudos to the Google team building it! https://skaffold.dev
-
Simplifying preview environments for everyone
To get a similar experience of preevy up, first we’ll need to split the build and deploy using process or alternatively employ tools that orchestrate build-tag-push-update-sync flow like Skaffold/Tilt.
- Is there a way to hot reload the code running in a container when I edit the codebase in VSCode?
-
Set up docker and kubernetes in ubuntu 22.04
We will be using docker and microk8s from Canonical. For running our software during development, we will be using skaffold which is a great tool developed by Google.
-
one container for a UI and one for express server. For dev would like to docker compose up. Couple questions
To add more context, if you are developing containers in a local dev environment, the minimum you should have is the Google Cloud SDK and Skaffold. The SDK will allow you to programmatically interact with Googleapis e.g. auth, services, resources. Skaffold will allow you to build and deploy to the cloud similar to working with a local dev environment.
-
How do you develop cloud-native applications locally on Kubernetes?
I have used both Skaffold and Devspace. I prefer the latter.
-
Launch HN: Moonrepo (YC W23) – Open-source build system
I wonder if it has some overlap with https://skaffold.dev/.
-
Building a RESTful API With Functions
K3d and Skaffold for local development
-
Does anyone else feel like this?
skaffold.dev - build in k8s - no more asking for the database password. All the plumbing to the backend is just done so it's easier for them to test and demo any branch
What are some alternatives?
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
argo-cd - Declarative Continuous Deployment for Kubernetes
trivy - Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
devspace - DevSpace - The Fastest Developer Tool for Kubernetes ⚡ Automate your deployment workflow with DevSpace and develop software directly inside Kubernetes.
dud - A lightweight CLI tool for versioning data alongside source code and building data pipelines.
okteto - Develop your applications directly in your Kubernetes Cluster
beneath - Beneath is a serverless real-time data platform ⚡️
telepresence - Local development against a remote Kubernetes or OpenShift cluster
typhoon-orchestrator - Create elegant data pipelines and deploy to AWS Lambda or Airflow
helm - The Kubernetes Package Manager
tsuru - Open source and extensible Platform as a Service (PaaS).
flux2 - Open and extensible continuous delivery solution for Kubernetes. Powered by GitOps Toolkit.