pachyderm
Flake8
pachyderm | Flake8 | |
---|---|---|
8 | 33 | |
6,077 | 3,263 | |
0.2% | 1.0% | |
9.8 | 7.3 | |
6 days ago | 8 days ago | |
Go | Python | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pachyderm
-
Open Source Advent Fun Wraps Up!
20. Pachyderm | Github | tutorial
-
Exploring Open-Source Alternatives to Landing AI for Robust MLOps
Pachyderm specializes in creating compliance-focused pipelines that integrate with enterprise-level storage solutions.
-
Show HN: We scaled Git to support 1 TB repos
There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
- pachyderm: Data-Centric Pipelines and Data Versioning
-
Awesome list of VCs investing in commercial open-source startups
Pachyderm - License prevents competition.
-
Airflow's Problem
I was at Airbnb when we open-sourced Airflow, it was a great solution to the problems we had at the time. It's amazing how many more use cases people have found for it since then. At the time it was pretty focused on solving our problem of orchestrating a largely static DAG of SQL jobs. It could do other stuff even then, but that was mostly what we were using it for. Airflow has become a victim of its success as it's expanded to meet every problem which could ever be considered a data workflow. The flaws and horror stories in the post and comments here definitely resonate with me. Around the time Airflow was opensource I starting working on data-centric approach to workflow management called Pachyderm[0]. By data-centric I mean that it's focused around the data itself, and its storage, versioning, orchestration and lineage. This leads to a system that feels radically different from a job focused system like Airflow. In a data-centric system your spaghetti nest of DAGs is greatly simplified as the data itself is used to describe most of the complexity. The benefit is that data is a lot simpler to reason about, it's not a living thing that needs to run in a certain way, it just exists, and because it's versioned you have strong guarantees about how it can change.
[0] https://github.com/pachyderm/pachyderm
-
One secret tip for first-time OSS contributors. Shh! 🤫 don't tell anyone else
Here is a demo run of lgtm on pachyderm
- Dud: a tool for versioning data alongside source code, written in Go
Flake8
-
To Review or Not to Review: The Debate on Mandatory Code Reviews
Automating code checks with static code analysis allows us to enforce code styling effectively. By integrating tools into our workflow, we can identify errors at an early stage, while coding instead of blocking us at the end. For instance, flake8 checks Python code for style and errors, eslint performs similar checks for JavaScript, and prettier automatically formats code to maintain consistency.
-
Enhance Your Project Quality with These Top Python Libraries
Flake8. This library is a wrapper around pycodestyle (PEP8), pyflakes, and Ned Batchelder’s McCabe script. It is a great toolkit for checking your code base against coding style (PEP8), programming errors (like SyntaxError, NameError, etc) and to check cyclomatic complexity.
- Django Code Formatting and Linting Made Easy: A Step-by-Step Pre-commit Hook Tutorial
-
Enhancing Python Code Quality: A Comprehensive Guide to Linting with Ruff
Flake8 combines the functionalities of the PyFlakes, pycodestyle, and McCabe libraries. It provides a streamlined approach to code linting by detecting coding errors, enforcing style conventions, and measuring code complexity.
-
Which is your favourite or go-to YouTube channel for being up-to-date on Python?
He made yesqa and pyupgrade (among others), and also works on flake8. His main job is for https://sentry.io/.
-
The Power of Pre-Commit for Python Developers: Tips and Best Practices
repos: - repo: https://github.com/psf/black rev: 21.7b0 hooks: - id: black language_version: python3.8 - repo: https://github.com/PyCQA/flake8 rev: 3.9.2 hooks: - id: flake8
-
Is it considered rude to completely change the formatting of someone else's code when making a PR?
https://github.com/psf/black it’s a PEP8 compliant formatter for Python codebases. If you don’t like auto formatting files you can use https://github.com/PyCQA/flake8 it just lists out all of the style issues so you can fix them manually.
-
Ruff: one Python linter to rule them all
I have no stake in that, but my observation is that the actual discussion appears to have both supporters and detractors rather than overwhelming support. Either way, it has nothing to do with whether or not it is realistic to say that Ruff is the "one Python linter to rule them all".
- Improve your Django Code with pre-commit
What are some alternatives?
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
Pylint - It's not just a linter that annoys you!
trivy - Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more
black - The uncompromising Python code formatter [Moved to: https://github.com/psf/black]
dud - A lightweight CLI tool for versioning data alongside source code and building data pipelines.
autopep8 - A tool that automatically formats Python code to conform to the PEP 8 style guide.
beneath - Beneath is a serverless real-time data platform ⚡️
pylama - Code audit tool for python.
typhoon-orchestrator - Create elegant data pipelines and deploy to AWS Lambda or Airflow
autoflake - Removes unused imports and unused variables as reported by pyflakes
tsuru - Open source and extensible Platform as a Service (PaaS).
prospector - Inspects Python source files and provides information about type and location of classes, methods etc