ArduinoJson VS FlatBuffers

I'm using the library ArduinoJson to storage arts in the ESP8266 memory at JSON format. But, i'm have difficult to storage more than 4 drawings, possibly because i don't know how scale, optimize and manager the memory space.

/******************************************************************* A sample project for making a HTTP/HTTPS GET request on an ESP32 and parsing it from JSON It will connect to the given request, parse the JSON and print the body to serial monitor Parts: ESP32 Dev Board Aliexpress: * - https://s.click.aliexpress.com/e/_dSi824B Amazon: * - https://amzn.to/3gArkAY * * = Affilate If you find what I do useful and would like to support me, please consider becoming a sponsor on Github https://github.com/sponsors/witnessmenow/ Written by Brian Lough YouTube: https://www.youtube.com/brianlough Tindie: https://www.tindie.com/stores/brianlough/ Twitter: https://twitter.com/witnessmenow *******************************************************************/ // ---------------------------- // Standard Libraries // ---------------------------- #include #include // ---------------------------- // Additional Libraries - each one of these will need to be installed. // ---------------------------- #include // Library used for parsing Json from the API responses // Search for "Arduino Json" in the Arduino Library manager // https://github.com/bblanchon/ArduinoJson //------- Replace the following! ------ char ssid[] = "yourSSID"; // your network SSID (name) char password[] = "yourPassword"; // your network key // For Non-HTTPS requests // WiFiClient client; // For HTTPS requests WiFiClientSecure client; // Just the base of the URL you want to connect to #define TEST_HOST "api.openweathermap.org" // Root cert of server we are connecting to // Baltimore CyberTrust Root - Expires 12 May 2025 // (FYI, from a security point of view you should not trust certs from other people, including me!) /*const char *server_cert = "-----BEGIN CERTIFICATE-----\n" "MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ\n" "RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD\n" "VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX\n" "DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y\n" "ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy\n" "VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr\n" "mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr\n" "IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK\n" "mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu\n" "XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy\n" "dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye\n" "jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1\n" "BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3\n" "DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92\n" "9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx\n" "jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0\n" "Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz\n" "ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS\n" "R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp\n" "-----END CERTIFICATE-----\n"; */ void setup() { Serial.begin(115200); // Connect to the WiFI WiFi.mode(WIFI_STA); WiFi.begin(ssid, password); Serial.println(""); // Wait for connection while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.print("Connected to "); Serial.println(ssid); Serial.print("IP address: "); Serial.println(WiFi.localIP()); //-------- // Checking the cert is the best way on an ESP32 // This will verify the server is trusted. //client.setCACert(server_cert); // If you don't want to verify the server // Unlike the fingerprint method of the ESP8266 which expires frequently // the cert lasts years, so I don't see much reason to ever // use this on the ESP32 client.setInsecure(); } void makeHTTPRequest() { // Opening connection to server (Use 80 as port if HTTP) if (!client.connect(TEST_HOST, 443)) { Serial.println(F("Connection failed")); return; } // give the esp a breather yield(); // Send HTTP request client.print(F("GET ")); // This is the second half of a request (everything that comes after the base URL) client.print("/data/2.5/weather?q=London, UK&appid=[yourOpenWeatherAPI]&units=metric"); // %2C == , client.println(F(" HTTP/1.1")); //Headers client.print(F("Host: ")); client.println(TEST_HOST); client.println(F("Cache-Control: no-cache")); if (client.println() == 0) { Serial.println(F("Failed to send request")); return; } //delay(100); // Check HTTP status char status[32] = {0}; client.readBytesUntil('\r', status, sizeof(status)); if (strcmp(status, "HTTP/1.1 200 OK") != 0) { Serial.print(F("Unexpected response: ")); Serial.println(status); return; } // Skip HTTP headers char endOfHeaders[] = "\r\n\r\n"; if (!client.find(endOfHeaders)) { Serial.println(F("Invalid response")); return; } // This is probably not needed for most, but I had issues // with the Tindie api where sometimes there were random // characters coming back before the body of the response. // This will cause no hard to leave it in // peek() will look at the character, but not take it off the queue while (client.available() && client.peek() != '{') { char c = 0; client.readBytes(&c, 1); Serial.print(c); Serial.println("BAD"); } // // While the client is still availble read each // // byte and print to the serial monitor // while (client.available()) { // char c = 0; // client.readBytes(&c, 1); // Serial.print(c); // } //Use the ArduinoJson Assistant to calculate this: //StaticJsonDocument<192> doc; DynamicJsonDocument doc(1024); //For ESP32/ESP8266 you'll mainly use dynamic. DeserializationError error = deserializeJson(doc, client); if (!error) { float coord_lon = doc["coord"]["lon"]; float coord_lat = doc["coord"]["lat"]; JsonObject main = doc["main"]; float main_temp = main["temp"]; Serial.print("Temp: "); Serial.println(main_temp); Serial.print("Longitude: "); Serial.println(coord_lon); Serial.print("Latitude: "); Serial.println(coord_lat); } else { Serial.print(F("deserializeJson() failed: ")); Serial.println(error.f_str()); return; } } void loop() { makeHTTPRequest(); delay(1500); }

I've logged a github issue, but I'm not convinced I've asked the right questions as I've never had to do anything like this before - am I missing something obvious?

https://arduinojson.org

Also, if you want a nice way to encapsulate data, look at ArduinoJson (https://arduinojson.org/). All your information can be stored as easily identifiable strings in a JSON format. I use it all the time and it removes a lot of the mucking about with figuring out how to parse your data to/from file.

The format is simple JSON data and self explanatory, no binary coding, just simple text, which you can decode using https://arduinojson.org/.

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

Much more in the discussion of this particular issue onGitHub: flatbuffers:Reverse engineering #4258

Related but not Rust-specific: FlatBuffers, Cap'n Proto.

If json files are a concern because of space, you can always look into something like protobuffers or flatbuffers. But whatever you use, you should try to find a solution where you don't have to think about the actual serialization/deserialization of your objects, and can just concentrate on the data.

Or, as an alternative to protobuffers, there's also flatbuffers, which is lighter weight and needs less memory: https://google.github.io/flatbuffers/

As the title suggests, I am considering using Flatbuffers as a way to parse sensor data that has been stored in local datafiles. The project language is python.