s4
PDM
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
s4
-
Ask HN: Does (or why does) anyone use MapReduce anymore?
the idea of map reduce remains a good one.
there are a number of interesting innovations in streaming systems that followed, mostly around reducing latency, reducing batch size, and alternate failure/retry strategies.
even hadoop could be hard to debug when hitting a performance ceiling for challenging workloads. the streaming systems took this even further, spark being notorious for fiddle with knobs and pray the next job doesn’t fail after a few hours, again.
i played around with the thinnest possible map reduce stack a while back[1][2]. i wanted to understand the performance ceiling for different workloads without all the impenetrable layers of data bureaucracy. turns out modern network and cpu are really fast when you stop adding random software layers like lasagna.
i think the future of data, for serious workloads, is gonna be bespoke. the primitives are just too good now.
1. https://github.com/nathants/s4
2. https://github.com/nathants/bsv
-
How fast are Linux pipes anyway?
pipes are great. is the other process on another cpu or another machine? honestly who cares.
https://github.com/nathants/s4/blob/master/examples/nyc_taxi...
-
Learning Go as a Python Developer: The Good and the Bad
i dragged my feet on go for a long time. i also thought that skipping go and moving to rust was the play. a few years later, i still write python often, but i don’t build systems with it. python i now use like bash, to glue things together and automate random things. it’s a fantastic language and i will never drop it.
the verbosity of go is the biggest hurdle for a pythonista. the thought of giving up context managers, decorators, iterators, comprehensions, exceptions, coroutines, it’s unthinkable. in comparison go is ugly. your aesthetic mind screams in protest.
write go full time. dive in. as months pass, not only will those aesthetic objections fade, your mental model from python cleanly transforms to go. go is what mypy tried to be. the cost was aesthetic changes. the benefit is worth it.
the zen of python says if it’s easy to explain it might be a good idea. this is go, and it is.
i rebuilt a reasonably sized project from python[1] to go[2] over the last few years. i also have a system that i maintained both python[3] and go[4] implementations for, sharing a test suite in python.
go, like python, is fantastic. use both in whatever amount works for you. don’t read about them, build with them. you won’t regret it.
1. https://github.com/nathants/cli-aws/tree/bb78e529e7d1d3f95ac...
2. https://github.com/nathants/libaws
3. https://github.com/nathants/s4/tree/python
4. https://github.com/nathants/s4
- Ask HN: Have you created programs for only your personal use?
- Super Simple Storage Service (S4)
PDM
-
Implementing Quality Checks In Your Git Workflow With Hooks and pre-commit
# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint
-
Beginning Python: Project Management With PDM
PDM is a solution that allows for easy creation and management of python projects. Some of the key features that will improve the management of python projects include:
-
A question about good practice when using docker.
You'd need a proper dependencies management tool like PDM or Poetry to exhaustively resolve and lock down all the transitive dependencies if you want to have anything closed to reproducible build.
-
pdm-dotenv: Simplify Your Project's Environment Variable Management
Are you working on a Python project that uses pdm for dependency management and dotenv for local environment variable and secrets management? Do you find it frustrating when CLI tools like pgcli don't automatically pick up your .env file, forcing you to resort to npm install -g dotenv-cli? I've got a more convenient solution for you!
-
PEP 582 rejected - consensus among the community needed
I first learned about PDM from a blog post written by one of the PDM contributers. The post was about OOPifying argparse to allow for easy creation/modification of subcommands that exist as their own classes/files, and to avoid maintaining a single long script with an endless number of subparser.add_argument(...) lines.
-
PEP 704 – Require virtual environments by default for package installers
That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm
-
This Week In Python
pdm – A modern Python package and dependency manager
- Pdm: A modern Python dependency manager supporting the latest PEP standards
-
How does a virtual environment work?
pdm and PEP 582 enter the chat
-
Boring Python: Code Quality
I'm liking PDM for a while now. Quicker than Poetry and built according to the Python package spec in mind and not as an afterthought. While it was originally meant to work with PEP 582, it works with virtual environments too (now default).
https://github.com/pdm-project/pdm
What are some alternatives?
epanet-js - Model a water distribution network in JavaScript using the OWA-EPANET engine
Poetry - Python packaging and dependency management made easy
fastmod - A fast partial replacement for the codemod tool
conda - A system-level, binary package and environment manager running on all major operating systems and platforms.
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
pip-tools - A set of tools to keep your pinned Python dependencies fresh.
wsl-ssh-pageant - A Pageant -> TCP bridge for use with WSL, allowing for Pageant to be used as an ssh-ageant within the WSL environment.
pip - The Python package installer
ppp_thing - A poorly written, minimum viable PPPoE client with session handoff between redundant FreeBSD routers
Pipenv - Python Development Workflow for Humans.
hnrss - Custom, realtime RSS feeds for Hacker News
PyFlow - Visual scripting framework for python - https://wonderworks-software.github.io/PyFlow