hydra
edn
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
hydra
- Hydra – a Framework for configuring complex applications
-
Show HN: Hydra - Open-Source Columnar Postgres
Nice tool, only unfortunate name, consider changing it. Already very well know security tool named hydra https://github.com/vanhauser-thc/thc-hydra been around since 2001. Then facebook went ahead and named their config tool hydra https://github.com/facebookresearch/hydra on top of it. Like we get it, hydra popular mythology but we could use more original naming for tools
-
Show HN: Hydra 1.0 – open-source column-oriented Postgres
This looks really impressive, and I'm excited to see how it performs on our data!
P.S., I think the name conflicts with Hydra, the configuration management library: https://hydra.cc/
-
Best practice for saving logits/activation values of model in PyTorch Lightning
I've been trying to learn PyTorch Lightning and Hydra in order to use/create my own custom deep learning template (e.g. like this) as it would greatly help with my research workflow. A lot of the work I do requires me to analyse metrics based on the logits/activations of the model.
-
[D] Alternatives to fb Hydra?
However, hydra seems to have several limitations that are really annoying and are making me reconsider my choice. Most problematic is the inability to group parameters together in a multirun. Hydra only supports trying all combinations of parameters, as described in https://github.com/facebookresearch/hydra/issues/1258, which does not seem to be a priority for hydra. Furthermore, hydras optuna optimizer implementation does not allow for early pruning of bad runs, which while not a deal breaker is definitely a nice to have feature.
-
Show HN: Lightweight YAML Config CLI for Deep Learning Projects
Do you hate the fact that they don't let you return the config file: https://github.com/facebookresearch/hydra/issues/407
-
Config management for deep learning
I kind of built this due to frustrations with Hydra. Hydra is an end to end framework, it locks you into a certain DL project format, it decides logging, model saving and a whole host of things. For example Hydra can do the same config file overwriting that I allow but you have to store the config file with the name config.yaml inside a specific folder. On top of that hydra doesn’t let you return the config file from the main function so you have to put all the major logic in the main function itself (link), the authors claim this is by design. I can find Hydra useful for a mature less experimental project. But in my robotics and ML research, I like being able to write code where I want and integrating it how I want, especially when debugging for which I think this package is useful. TLDR; If you just want the config file functionality use my package, if you want a complete DL project manager use Hydra. While hydra implements this config file functionality, it also adds a lot of restrictions to project structure that you might not like.
-
The YAML Document from Hell
For managing configs of ML experiments (where each experiment can override a base config, and "variant" configs can further override the experiment config, etc), Hydra + Yaml + OmegaConf is really nice.
https://hydra.cc/
I admit I don't fully understand all the advanced options in Hydra, but the basic usage is already very useful. A nice guide is here:
https://florianwilhelm.info/2022/01/configuration_via_yaml_a...
- Hydra - namestitev in osnovna uporaba
- Hydra - namestitevt in osnovna uporaba
edn
-
Apple releases Pkl – onfiguration as code language
> was utterly surprised how no one ever apparently has thought to create a configuration/templating system that's basically a fancy library on top of Scheme.
There's Clojure's extensible data notation: https://github.com/edn-format/edn
- Why the fuck are we templating YAML? (2019)
-
I made a basic python client and ORM for XTDB
A thin language layer around edn/datalog, the query language
-
What Is Wrong with TOML?
EDN (Extensible Data Notation) is a subset of Clojure: https://github.com/edn-format/edn
It is:
- Streamable
- Extensible
- Whitespace-insensitive, but there are formatting conventions for readability
-
The real reason JSON has no comments
To begin with, EDN is somewhat like the JSON of Clojure. And regarding the code is data/data is code nature of Clojure, it is Clojure. It doesn't have some of the vagaries of JSON, and it is also extensible.
-
Ron: Rusty Object Notation
Alien is not a reason something is bad, just that's it's unusual. JSON was a bit alien when it first arrived as well, as everyone was used to XML at the time.
`{num 5, val 4}` looks fine to me, but we can do even better! We already know objects/maps are always in pairs, so we don't really need that comma either. Just do `{num 5 val 4}` and we save yet another unnecessary characters.
Of course, I didn't come up with this format myself, what I actually want JSON to be is EDN (https://github.com/edn-format/edn) which is a standalone format but also directly used in Clojure, so it already exists inside a programming language and works very well. There keys are strings though, so you example would end up being `{"num" 5 "val" 5 "person" var}`, where commas are optional.
-
JSON vs. XML with Douglas Crockford
I just checked out the spec, and it gets pretty ugly in the Table section. A lot of the json examples are both shorter and IMO more precise. Stuff that’s not allowed with [table] is allowed with [[table]], and it’s confusing to understand what level of depth I’m at.
I’ll take edn over any of “em. https://github.com/edn-format/edn
-
Taming the Time: how to install & develop with XTDB
As XT is written in Clojure and it natively supports Clojure’s data types, we were not satisfied with available JSON types and decided to give EDN a try - that way we would have way more supported types:
-
Design patterns are a solution to the problem OOP itself creates
Compare the nightmare that is pickling with how simple it is to serialize pure data with edn in clojure. What ends up happening is people passing around JSONs or whatever and writing parsing/encoding code at each end, which makes things unnecessarily more complex, and dangerous, and error prone, and boring, etc...
- The YAML Document from Hell
What are some alternatives?
dynaconf - Configuration Management for Python ⚙
json - JSON for Modern C++
ConfigParser
EPOE-Forked - Github repository for EPOE-Forked
python-dotenv - Reads key-value pairs from a .env file and can set them as environment variables. It helps in developing applications following the 12-factor principles.
jq - Command-line JSON processor [Moved to: https://github.com/jqlang/jq]
python-decouple - Strict separation of config from code.
yamllint - A linter for YAML files.
django-environ - Django-environ allows you to utilize 12factor inspired environment variables to configure your Django application.
dhall-lang - Maintainable configuration files
classyconf - Declarative and extensible library for configuration & code separation
json - A tested JSON parser / serializer