dogma VS go-concise-encoding

Yes, the offset function does this by specifying an offset to branch to. For example the ICO dir_entry, which is a directory list of icon resources in the file.

- https://github.com/kstenerud/dogma/blob/master/v1/examples/i...

I'm facing this dilemma now with the Dogma metalanguage [1]. On the one hand, I want it to be small so that it can be picked up quickly and mastered easily (very important since its primary use case is documentation). On the other hand, it needs enough power to actually be useful in the real world.

In the end I opted to avoid most conveniences since they can easily be built using a few common macros such as `u16(v) = ordered(uint(16,v))` and the like, except for cases where it would be just silly not to (like strings and regex-style repetition chars ? * +).

But even then the spec is over 2000 lines long (admittedly the examples take up a fair bit of room, though).

[1] https://github.com/kstenerud/dogma/blob/master/v1/dogma_v1.m...

Hi HN!

I've just released the 5th beta of the Dogma metalanguage, which is designed to describe text and binary data for use in documentation.

https://github.com/kstenerud/dogma/blob/master/v1/dogma_v1.md

Most of the kinks are now worked out, and I'm hoping for a final release soon.

Notable changes:

* Cleaner byte ordering support with new byte_order, ordered, and bom_ordered functions

* New peek and offset functions

* Renamed swapped function to reversed

* More examples in the examples dir https://github.com/kstenerud/dogma/tree/master/v1/examples

Thanks to all for reviewing and helpful criticism :)

The diversity of notations stems from the fact that the current BNF notations are always lacking in something fundamental that each use case needs.

For me, the breaking point came in trying to describe binary formats. I eventually had to come up with my own: https://github.com/kstenerud/dogma

https://concise-encoding.org/ will be released this year and has been in development for 5 years. This one has been slow because of all the support projects such as https://github.com/kstenerud/compact-float and https://github.com/kstenerud/compact-time and https://github.com/kstenerud/enctool

https://github.com/kstenerud/dogma has been in development for 4 months and will be released in about a month. This is yet another support project for Concise Encoding (hopefully the last!).

https://github.com/kstenerud/kscrash was in development for about a year and a half before being released.

https://github.com/kstenerud/Musashi took just under a year before its first release as a MAME core (after a TON of testing - I spent 4x more time testing it than I did writing it).

But other smaller things that don't require so much precision I just write up in a couple of weeks and release, such as https://github.com/kstenerud/virtual-builders

The key is to build empathy with your potential users. What will their motivations be that lead them to try out your project? What will they be looking for when they use it? What would someone who has never seen the project before struggle with? (that last one is the hardest because you're so close to the project that it's hard to see what it's like to know nothing about it). These tell you what kind of UX you'll need, what kind of documents you'll need, what kind of tutorials you'll need, how to structure your project for your target audience, etc ("it's for everybody" is never a good idea).

A lot of times it comes down to recruiting people to just try it and do a brain dump of everything that goes wrong or frustrates or confuses them. I've done show-HN posts for things that are unreleased, just to get the valuable criticism. It's nearly impossible to develop good projects in a vacuum.

If you're not posting out of worry for doing it wrong, THAT is doing it wrong. The point is to find out where you're doing it wrong so that you can correct it! And that's where the crowd is a HUGE help.

I have two projects that could use help:

---

The first is Concise Encoding, a secure ad-hoc data format (think JSON, but secure, more data types, and also has a binary encoding). https://concise-encoding.org/

The spec is largely done, and I'm primarily working on the reference implementation (in golang) and portable tests (so people don't have to keep rewriting tests for every implementation), but I could use help getting other language implementations out there.

---

The second one is Dogma, a BNF-style metalanguage for describing binary and text data (because there wasn't a decent one for describing binary data). https://github.com/kstenerud/dogma

Dogma is mostly there as well, but while it is primarily focused around writing technical documentation (and I have written a syntax highlighter for VS Code), there are probably a lot of other tools that could be built around this technology (which I simply don't have enough time to pursue).

Two things, one which supports the other:

First is Concise Encoding, a secure, binary and text ad-hoc data format with full type support: https://concise-encoding.org/

Almost all existing data formats are horribly insecure and ripe for exploitation (many of them are already). And they can't be fixed because the formats aren't versioned.

-----------------

But describing the data format was painful because most existing metalanguages are only for describing textual grammars, and the few that support binary aren't really suited for documenting for human consumption.

So I'm putting the finishing touches on Dogma ( https://github.com/kstenerud/dogma/blob/master/dogma_v1.md ), a modernized BNF-style metalanguage with better expressiveness and binary grammar support.

For example:

A UDP packet consists of a source port, a destination port, a length, a checksum, and a body. The length field refers to the size of the entire packet, not just the body.

    udp_packet   = src_port

I'm not sure, TBH... Maybe I'm just a glutton for punishment?

I'm currently partway through refactoring all of the portable unit tests for Concise Encoding ( https://github.com/kstenerud/go-concise-encoding/tree/master... ) and it is a SLOG! I mean, so goddamn boring and tedious that I wanna stick an ice pick through my skull. There's easily another 200 hours of this terrible work ahead and I'll be right back to it the moment Dogma v1 is published in a few weeks (Dogma has been kind of a vacation from it in a lot of ways).

Do I dread it? Yes. Am I still going to do it? Yes. Because at the end of the day I want to be able to stand back and say "I made that. I completed it - ALL of it. It's not perfect, but it's doing its job and people are using it."

It's still in the prerelease stage, but v1 will be released later this year. I'm mostly getting hits from China since they tend to be a lot more worried about security. I expect the rest of the world to catch on to the gaping security holes of JSON and friends in the next few years as the more sophisticated actors start taking advantage of them. For example https://github.com/kstenerud/concise-encoding/blob/master/ce...

There are still a few things to do:

- Update enctool (https://github.com/kstenerud/enctool) to integrate https://cuelang.org so that there's at least a command line schema validator for CE.

- Update the grammar file (https://github.com/kstenerud/concise-encoding/tree/master/an...) because it's a bit out of date.

- Revamp the compliance tests to be themselves written in Concise Encoding (for example https://github.com/kstenerud/go-concise-encoding/blob/master... but I'll be simplifying the format some more). That way, we can run the same tests on all CE implementations instead of everyone coming up with their own. I'll move the test definitions to their own repo when they're done and then you can just submodule it.

I'm thinking that they should look more like:

    c1

I'll be working on the reference implementation [1] of Concise Encoding [2], which is a secure data format for the modern world. My aim is to replace insecure and clunky formats like JSON and XML and the various binary formats that do similar things less conveniently.

In a nutshell:

- Existing ad-hoc formats are too loosely defined to be secure, and that's becoming a huge problem as the bad guys become more sophisticated. CE is tightly specified and designed to mitigate exploitation of codecs.

- CE is a twin text and binary format. Humans view and edit in text, and machines send it in binary, so you get the convenience of text and the efficiency of binary for free.

- CE supports the fundamental types natively. Stringifying is buggy, causes incompatibilities, and opens security holes. And it's completely unnecessary with a properly designed data format.

[1] https://github.com/kstenerud/go-concise-encoding

[2] https://concise-encoding.org/

I'm building a general-purpose data format for the modern age. The old ones are too bulky, too insecure, and too limiting.

* Secure: As a tightly specified format, Concise Encoding doesn't suffer from the security problems that the more loosely defined formats do. Everything is done one way only, leaving less of an attack surface.

* Efficient: As a twin binary/text format, Concise Encoding retains the text-based ease-of-use of the old text formats, but is stored and transmitted in the simpler and smaller binary form, making it more secure, easier on the energy bill, and easier on the planet.

* Versatile: Supports all common types natively. 90% of users won't need any form of customization.

* Future-proof: As a versioned format, Concise Encoding can respond to a changing world without degenerating into deprecations and awkward encodings or painting itself into a corner.

* Plug and play: No extra compilation steps or special description formats or crazy boilerplate.

https://concise-encoding.org

Reference implementation (golang): https://github.com/kstenerud/go-concise-encoding

One thing golang did right is the go playground. When I put code in my README, I also include a playground link.

Example: https://github.com/kstenerud/go-concise-encoding#library-usa...

I could use some help if you're interested in implementing an ad-hoc data format codec in different languages.

https://concise-encoding.org is nearing release, but building the reference implementation (https://github.com/kstenerud/go-concise-encoding) has taken so much of my spare time that I couldn't even think about other languages (especially since I'll need to focus on the schema format next, and the proto-RPC protocol after that).

I've started doing this in my larger projects e.g. https://github.com/kstenerud/go-concise-encoding/blob/master...

An architecture document should be the code equivalent of a combined street map and tourist guide. Its purpose is to bring strangers up to a minimum level of familiarity with the code as quickly as possible. That includes where things are, why it was architected this way, things to look out for, and a few interesting points of weirdness perhaps.

Concise Encoding: https://concise-encoding.org

The friendly data format for human and machine. Think JSON, but with 1:1 compatible twin binary and text formats and rich type support.

* Edit text, transmit binary. Humans love text. Machines love binary. With Concise Encoding, conversion is 1:1 and seamless.

* Rich type support. Boolean, integer, float, string, bytes, time, URI, UUID, list, map, markup, metadata, etc.

* Plug and play. No schema needed. No special syntax files. No code generation. Just import and go.

I'm in the process of finishing up the reference implementation (https://github.com/kstenerud/go-concise-encoding), after which I'll start on the schema specification. Once that's done, I have a low-level communication protocol that will use this format under the hood.