codec-from-scratch
cbor
Our great sponsors
codec-from-scratch | cbor | |
---|---|---|
5 | 4 | |
244 | 659 | |
- | - | |
2.0 | 8.6 | |
about 1 year ago | 5 days ago | |
Go | Go | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
codec-from-scratch
-
Video Codec in 100 lines of Rust
I discovered this while looking for more resources for my own codec from scratch project: https://github.com/kevmo314/codec-from-scratch
-
QOIR: A fast, simple, lossless image file format
> For example, if you have a relatively wimpy camera that streams out a 160×120 grayscale image, it might be fun to see how far you can push something like combining:
I'm typically not one for self-promo but I did this with video and you can get 90% compression with exactly those three basic steps. RGB -> YUV420, delta encoding, then basically-upgraded-RLE. You can find the guide here: https://github.com/kevmo314/codec-from-scratch
- Video Encoding from Scratch
cbor
-
Rob Pike: Gobs of data (2011)
Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.
By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).
There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.
--
1: https://github.com/alecthomas/go_serialization_benchmarks
2: https://github.com/golang/go/issues/29766#issuecomment-45492...
--
3: https://github.com/goccy/go-json
4: https://github.com/vmihailenco/msgpack
5: https://github.com/fxamacker/cbor
--
6: https://github.com/valyala/fasthttp#faq
--
7: https://github.com/tidwall/gjson
8: https://github.com/tidwall/sjson
-
What is the fastest way to encode the arbitrary struct into bytes?
Ha, no suggestions for cbor https://github.com/fxamacker/cbor
- How can we umarshal a Big JSON effectively?
-
80x improvements in caching by moving from JSON to gob
If you need to support non-Go code I suggest you explore CBOR. There are a few implementations in Go but his is my favorite.
What are some alternatives?
gamut - Image encoding and decoding library for D. Detailed layout control. Experimental codec QOIX.
asn1
jcodec - JCodec main repo
goprotobuf - Go support for Google's protocol buffers
less-avc - less Advanced Video Coding (H.264) encoding
gogoprotobuf - [Deprecated] Protocol Buffers for Go with Gadgets
go-capnproto - Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See https://github.com/zombiezen/go-capnproto2 for 2.0 which has rpc and capabilities.
go-codec - idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]
jsoniter - A high-performance 100% compatible drop-in replacement of "encoding/json"
fwencoder - Fixed width file parser (encoder/decoder) in GO (golang)
mapstructure - Go library for decoding generic map values into native Go structures and vice versa.
pletter - A standard way to wrap a proto message