hydra
jsonnet
hydra | jsonnet | |
---|---|---|
14 | 48 | |
8,229 | 6,762 | |
1.6% | 0.5% | |
6.3 | 8.4 | |
22 days ago | 8 days ago | |
Python | Jsonnet | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
hydra
- Hydra – a Framework for configuring complex applications
-
Show HN: Hydra - Open-Source Columnar Postgres
Nice tool, only unfortunate name, consider changing it. Already very well know security tool named hydra https://github.com/vanhauser-thc/thc-hydra been around since 2001. Then facebook went ahead and named their config tool hydra https://github.com/facebookresearch/hydra on top of it. Like we get it, hydra popular mythology but we could use more original naming for tools
-
Show HN: Hydra 1.0 – open-source column-oriented Postgres
This looks really impressive, and I'm excited to see how it performs on our data!
P.S., I think the name conflicts with Hydra, the configuration management library: https://hydra.cc/
-
Best practice for saving logits/activation values of model in PyTorch Lightning
I've been trying to learn PyTorch Lightning and Hydra in order to use/create my own custom deep learning template (e.g. like this) as it would greatly help with my research workflow. A lot of the work I do requires me to analyse metrics based on the logits/activations of the model.
-
[D] Alternatives to fb Hydra?
However, hydra seems to have several limitations that are really annoying and are making me reconsider my choice. Most problematic is the inability to group parameters together in a multirun. Hydra only supports trying all combinations of parameters, as described in https://github.com/facebookresearch/hydra/issues/1258, which does not seem to be a priority for hydra. Furthermore, hydras optuna optimizer implementation does not allow for early pruning of bad runs, which while not a deal breaker is definitely a nice to have feature.
-
Show HN: Lightweight YAML Config CLI for Deep Learning Projects
Do you hate the fact that they don't let you return the config file: https://github.com/facebookresearch/hydra/issues/407
-
Config management for deep learning
I kind of built this due to frustrations with Hydra. Hydra is an end to end framework, it locks you into a certain DL project format, it decides logging, model saving and a whole host of things. For example Hydra can do the same config file overwriting that I allow but you have to store the config file with the name config.yaml inside a specific folder. On top of that hydra doesn’t let you return the config file from the main function so you have to put all the major logic in the main function itself (link), the authors claim this is by design. I can find Hydra useful for a mature less experimental project. But in my robotics and ML research, I like being able to write code where I want and integrating it how I want, especially when debugging for which I think this package is useful. TLDR; If you just want the config file functionality use my package, if you want a complete DL project manager use Hydra. While hydra implements this config file functionality, it also adds a lot of restrictions to project structure that you might not like.
-
The YAML Document from Hell
For managing configs of ML experiments (where each experiment can override a base config, and "variant" configs can further override the experiment config, etc), Hydra + Yaml + OmegaConf is really nice.
https://hydra.cc/
I admit I don't fully understand all the advanced options in Hydra, but the basic usage is already very useful. A nice guide is here:
https://florianwilhelm.info/2022/01/configuration_via_yaml_a...
- Hydra - namestitev in osnovna uporaba
- Hydra - namestitevt in osnovna uporaba
jsonnet
-
A Reasonable Configuration Language
jsonnet[1] and kapitan[2] are the tools I currently use. Their learning curve is not optimal (and I tried to contribute to smoothen it with a jsonnet course[3] and a 'get started wit kapitan' blog post[4]), but once used to it it's hard to do without, and their combination makes them even more useful (esp. if you deploy K8s).
In Ruud's case, Jsonnet might have been worth looking at as Hashicorp tools can be configured with json in addition to HCL. But that would have been less fun I guess ;-)
I hope for Ruud it finds its niche, there's quite some competition in this field!
1: https://jsonnet.org/
-
Pkl, a Programming Language for Configuration
Kubernetes config is a decent example. I had ChatGPT generate a representative silly example -- the content doesn't matter so much as the structure:
https://gist.github.com/cstrahan/528b00cd5c3a22e3d8f057bb1a7...
Now consider 100s (if not 1000s) of such files.
I haven't given Pkl an in depth look yet, but I can say that the Industry Standard™ of "simple YAML" + string substitution (with delicate, error prone indentation -- since YAML is indentation sensitive) is easily beat by any of:
- https://jsonnet.org/
- https://nickel-lang.org/
- https://nixos.org/manual/nix/stable/language/index.html
- https://dhall-lang.org/
- (insert many more here, probably including Pkl)
-
Introduction to Jsonnet: The YAML/JSON templating language
jsonnet cli: link
-
10 Ways for Kubernetes Declarative Configuration Management
Jsonnet: A data template language implemented in C++, suitable for application and tool developers, can generate configuration data and organize, simplify and manage large configurations without side effects.
-
-❄️- 2023 Day 4 Solutions -❄️-
[Language: Jsonnet] (on GitHub)
-
What Is Wrong with TOML?
Maybe you'd like jsonnet: https://jsonnet.org/
I find it particularly useful for configurations that often have repeated boilerplate, like ansible playbooks or deploying a bunch of "similar-but" services to kubernetes (with https://tanka.dev).
Dhall is also quite interesting, with some tradeoffs: https://dhall-lang.org/
A few years ago I did a small comparison by re-implementing one of my simpler ansible playbooks: https://github.com/retzkek/ansible-dhall-jsonnet
- Show HN: Keep – GitHub Actions for your monitoring tools
-
That people produce HTML with string templates is telling us something
Apologies for the lack of context, and for missing this comment until today.
Both are tools for defining kubernetes manifests (which are YAML) in a reusable manner.
Jsonnet is a formally specified extension of JSON. It’s essentially a functional programming language (w/some object oriented features) that generates config files in JSON/YAML/etc, so it’s straightforward to determine whether an input file is valid, and to throw an error that points to an exact line if it’s not. It has a high learning curve, especially for people whose only experience is with imperative languages.
https://jsonnet.org/
Helm charts also generate YAML/JSON config files, but they use Go templating. This is easier and faster to understand, since it’s mostly string substitution and not much logic (there’s conditionals, iterators, and very basic helper functions). Unfortunately a simple typo or mistake can cause errors that are difficult to diagnose (the message may indicate a problem far away in code from the actual mistake). It can also generate output that’s valid according to the string templating rules, but not what was intended, which can be very confusing to debug.
Despite these shortcomings, the vast majority of kubernetes applications are distributed as helm charts. I understand why things ended up this way, but I still wish it were more common for people to invest the upfront effort to learn the superior tool, so it could be more widespread.
-
TOML: Tom's Obvious Minimal Language
I like Google's Jsonnet [1], which has all of this except for 4.
Jsonnet is quite mature, with fairly wide language adoption, and has the benefit of supporting expressions, including conditionals, arithmetic, as well as being able to define reusable blocks inside function definitions or external files.
It's not suitable as a serialization format, but great for config. It's popular in some circles, but I'm sad that it has not reached wider adoption.
[1] https://jsonnet.org/
- Jsonnet – The Data Templating Language
What are some alternatives?
dynaconf - Configuration Management for Python ⚙
kube-libsonnet - Bitnami's jsonnet library for building Kubernetes manifests
ConfigParser
dhall-lang - Maintainable configuration files
python-dotenv - Reads key-value pairs from a .env file and can set them as environment variables. It helps in developing applications following the 12-factor principles.
cue - CUE has moved to https://github.com/cue-lang/cue
python-decouple - Strict separation of config from code.
cue - The home of the CUE language! Validate and define text-based and dynamic configuration
django-environ - Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.
json5 - JSON5 — JSON for Humans
classyconf - Declarative and extensible library for configuration & code separation
cdk8s - Define Kubernetes native apps and abstractions using object-oriented programming