Directly memory export/import

• goMemExport

  1 2 10.0 Go

  Example go project to export and import go memory directly without any encoder. It is fast.

Hi. I was searching fastest way to import big-size data in init-time. Firstly we were not caring encoder. We were thinking "Json is enough for anything". However json mean parsing and parsing mean extra time and extra source usage. Then we decided to prepare some "preparer" software to prepare data for golang memory directly. This preparer software is parsing some international data from network, deserializing into struct and exporting struct memory to linear memory. (In our case it is []byte). Finally we are exporting this prepared data into a file. On host server, we are reading this prepared exported data with embed and finally converting directly without any decoder to our struct. Of course with help unsafe and reflect. Export process's speed is not important to us however import is. I saw approximately 200-400 nanoseconds on benchmarks. Now, I need review what we did and is it healthy. Thank you very much. Here example repository https://github.com/raifpy/goMemExport

• go-capnp

  3 1,157 6.3 Go

  Cap'n Proto library and code generator for Go

  

Use Cap'n Proto (https://github.com/capnproto/go-capnproto2) or flatbuffers.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project