Rob Pike: Gobs of data (2011)

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com
Golang Go JSON Serialization Networking

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • gerbil-cbor

    Discontinued CBOR/RFC 8949 implementation for Gerbil Scheme

    • I'm in the same boat. Not to mention security concerns that often crop up in (interpreted) language specific deserialization (I'm looking at you, pickle, thinly veiled `eval()`). I agree that CBOR should generally be the serialization tool of choice for self-describing data (ie. in places where you might otherwise choose JSON).

    And if your language of choice doesn't have a CBOR lib, CBOR is fairly easy to implement and writing a encoder/decoder is very fun! I recently completed my implementation for the Gerbil Scheme language last week [0].

    [0]: https://github.com/chiefnoah/gerbil-cbor

  • ion-docs

    Source for the GitHub Pages for Ion.

    • Have you tried the superset of json from AWS?

    https://amazon-ion.github.io/ion-docs/

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • go_serialization_benchmarks

    Benchmarks of Go serialization methods

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • go

    The Go programming language

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • go-json

    Fast JSON encoder/decoder compatible with encoding/json for Go

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • msgpack

    msgpack.org[Go] MessagePack encoding for Golang (by vmihailenco)

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • cbor

    CBOR codec (RFC 8949) with CBOR tags, Go struct tags (toarray, keyasint, omitempty), float64/32/16, big.Int, and fuzz tested billions of execs.

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • fasthttp

    Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • GJSON

    Get JSON values quickly - JSON parser for Go

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

  • sjson

    Set JSON values very quickly in Go

    • Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

    By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

    There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

    --

    1: https://github.com/alecthomas/go_serialization_benchmarks

    2: https://github.com/golang/go/issues/29766#issuecomment-45492...

    --

    3: https://github.com/goccy/go-json

    4: https://github.com/vmihailenco/msgpack

    5: https://github.com/fxamacker/cbor

    --

    6: https://github.com/valyala/fasthttp#faq

    --

    7: https://github.com/tidwall/gjson

    8: https://github.com/tidwall/sjson

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts