go-json VS Task

For go -> you can actually get away with the standard json encoding package. Or if you want a slightly better one, I prefer goccy/go-json

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

--

1: https://github.com/alecthomas/go_serialization_benchmarks

2: https://github.com/golang/go/issues/29766#issuecomment-45492...

--

3: https://github.com/goccy/go-json

4: https://github.com/vmihailenco/msgpack

5: https://github.com/fxamacker/cbor

--

6: https://github.com/valyala/fasthttp#faq

--

7: https://github.com/tidwall/gjson

8: https://github.com/tidwall/sjson

Drop-in replacement for the stdlib JSON package: https://github.com/goccy/go-json

go stdlib json encoding/decoding is incredibly slow, not sure for how much longer because there are drop in replacements now that I think are just as strict and feature parity.

3rd party JSON libraries could help if you were comparing JSON. https://github.com/goccy/go-json

However, I understand the code that is generated is super optimized. For example, rather than use a router, it does a static code generated router. Rather than use goccy/go-json, it does manual marshalling.

Minimal dependencies: julienschmidt/httprouter and goccy/go-json

I suggested using https://github.com/goccy/go-json at my work, since its a drop in replacement for the standard lib, but there are people who apprehensive. In my opinion the performance gains are significant to justify adoption. But I'd like your input.

go-json for encoding/decoding

So many tools in this space! This one looks a little bit like go-task, but it seems maybe better for production workflows because if timeout support, while go-task seems more aimed to command line work/makefile replacement.

—-

https://github.com/go-task/task

View on GitHub

A similar tool is `task` https://taskfile.dev/ . It is quite capable and also a single executable. I've grown to quite like it.

check out tasks - a bit of a learning curve but arguably more powerful imo

That's when I came across taskfile.dev. Task is an automation tool designed to be more accessible than other options, such as GNU Make.

Full disclosure, I did not review Conda or Hatch fully. Not that there is anything explicitly wrong with either of them. Conda is too specific to the scientific community for my general taste. Hatch seems to go well with Conda and also uses the PyProject manifest as well. It's nice that it gives you several built in tools, similar to commit hooks, but I tend to like to roll my own via a Taskfile and run them with Poetry.

Taskfile is a tool for streamlining repetitive development tasks. It helps automate activities like building, testing, and deploying applications. Unlike Makefile, Taskfile uses YAML for configuration, making it more readable and user-friendly.

9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest)

This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components.

My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic.

My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not).

[0] https://taskfile.dev/

task - a task runner and a replacement for make