fasthttp VS m3o

Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.

By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).

There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.

--

1: https://github.com/alecthomas/go_serialization_benchmarks

2: https://github.com/golang/go/issues/29766#issuecomment-45492...

--

3: https://github.com/goccy/go-json

4: https://github.com/vmihailenco/msgpack

5: https://github.com/fxamacker/cbor

--

6: https://github.com/valyala/fasthttp#faq

--

7: https://github.com/tidwall/gjson

8: https://github.com/tidwall/sjson

Fasthttp is one of the most powerful webservers written in Go, I'm working on a project that makes it possible to use it as a webserver for Python.

IDE: use whatever make you productive. I personally use vscode. VCS: git, as golang communities use github heavily as base for many libraries. AFAIK Linter: use staticcheck for linting as it looks like mostly used linting tool in go, supported by many also. In Vscode it will be recommended once you install go plugin. Libraries/Framework: actually the standard libraries already included many things you need, decent enough for your day-to-day development cycles(e.g. `net/http`). But here are things for extra: - Struct fields validator: validator - Http server lib: chi router , httprouter , fasthttp (for non standard http implementations, but fast) - Web Framework: echo , gin , fiber , beego , etc - Http client lib: most already covered by stdlib(net/http), so you rarely need extra lib for this, but if you really need some are: resty - CLI: cobra - Config: godotenv , viper - DB Drivers: sqlx , postgre , sqlite , mysql - nosql: redis , mongodb , elasticsearch - ORM: gorm , entgo , sqlc(codegen) - JS Transpiler: gopherjs - GUI: fyne - grpc: grpc - logging: zerolog - test: testify , gomock , dockertest - and many others you can find here

The best example I can give you is https://github.com/nutsdb/nutsdb it’s great project that got me started, one thing one should know is Go is different “yep” so there’re some coding habits that may bite you in Go and the Go compiler won’t correct you, you wanna learn about optimizations, unsafe usage check out https://github.com/valyala/fasthttp (note this is deep the rabbit hole), wanna learn concurrency check out ants https://github.com/panjf2000/ants with a little aid from “Go by example” you’re good to go

As I said in another comment, I think net/http is a good cautionary tale here. It was designed to be easy to use, and then grew organically, but performance never seems to have been a goal. fasthttp solves this, but bifurcates the ecosystem and passes on those costs to everyone who uses it. If net/http had been designed with performance in mind, this could have been avoided. net/http can't be removed or optimized, so this is a situation the Go ecosystem is effectively stuck with forever. At best, a faster version may end up in the std lib, just like netip is more modern and faster than net but the ecosystem is still bifurcated and adoption of the new package has been slow.

Fiber, while has a lot of batteries included and decent for many use cases, is known for having corner cases (because of internals like fasthttp) like https://github.com/valyala/fasthttp/issues/622

Sorry what I meant is I've been working on the open source Micro project for 8 years which underpins this. The chat app itself was not really something I meant as being open source but yes it's in a separate repo with the API hosting product I built called M3O.com.

https://github.com/m3o/m3o

Email: [email protected]

Spent the last 10 years mostly working with microservices and Go based startups, although I would not recommend microservices to most companies. I can save you a few million dollars if you wonder why.

I'm most passionate about improving DevEx in companies. Things like writing custom ORMs for lesser known/supported databases (see eg. https://github.com/gocassa/gocassa). Mostly worked with startups from $2M-$500M funding range, with the occasional enterprise gig.

Used to run a chicken shop as a hobby project which made me careful of accepting management positions, haha, people are hard. I would love to be a product owner, I mostly designed the https://m3o.com/ product with the CEO recently and implemented the MVP of it. It's open source stuff, check it out https://github.com/m3o

Currently working for a US startup but my contract is ending soon.

Cheers!

It goes back to, what is a framework and what qualifies as a big framework here. I think classic rails isn't the fit, but something that's an extension of gRPC definitely works. What gets handcrafted is a lot of layers around gRPC or far more stuff around HTTP.

When I'm working on personal projects, frameworks don't make sense for me. When I'm trying to engineer something at scale e.g https://m3o.com then I need that standardisation at the platform layer, the framework layer, the API layer.

What if any is the relationship between https://m3o.com/ and https://micro.dev/ ?

Yup, I was the author of go-micro. It was very much a standalone Go framework aka common interfaces grouped together for distributed systems development. By using interfaces they effectively became pluggable abstractions for infrastructure. Unfortunately I don't think a Go library alone solves the problems I was trying to solve so it got merged into Micro which is platform that includes a CLI, API, Runtime, etc. It powers https://m3o.com

We can potentially do this on https://m3o.com. It doesn't exist yet but would make sense as the next service we offer. Further details would be useful e.g endpoints required.

We ended up building this in-house like most mentioned. Speakeasy and Stainless are productizing it. Our goal was just to make it available for APIs we were offering to others (https://m3o.com).

I value your points because they are the same concerns I would have being a CTO of a company. Ultimately vendors don't yet care enough about this problem to invest in it and I think ultimately that's a mistake because while we have standardisation at the Cloud infrastructure layer, we're missing everything above it. The cost of development to an organisation in these services is quite frankly astronomical. You've got hundreds of devs rewriting identical CRUD services or proxy shims to existing SaaS across the entire industry. That's millions in capex just being burned.

In relation to being a CTO of a small startup, yea OSS maintainer risk is tough. You want to use projects that are used by hundreds of companies and actively maintained. In my case, I am the primary maintainer and it's used for a cloud service called M3O - https://m3o.com. I think it will take a while before we're in a place to warrant more buy in but my hope is eventually it'll get there or at the very least people will come to use the APIs serviced by M3O.

On architecture, I mean you're quite literally talking about software "build vs buy" tradeoffs for the entirety of all software you ever write. In this case, do I integrate something else or write it myself. I think that comes down to the same assessment of whether you should offload to some other piece of software versus your own. When it comes to domain specific services this is always tough yet we see the adoption of the likes of Twilio for SMS, Sendgrid for Email and Stripe for Payments so I'd argue we're getting closer to blurring the lines now.

On cost, you can use the hosted offering - https://m3o.com - but at this point the reason I'm sharing the open source services is really because I think that adoption curve to a cloud service takes a lot longer especially with domain specific services. I would argue these services while on the surface appear a commodity, the development time and integration cost of using bespoke independent APIs or services has a much higher cost. Everyone internally ends up writing proxy/shims to SaaS products to try eliminate this risk for themselves. I just think we should standardise a lot of our business logic service consumption.

lol, really don't want to be in the same sentence as Urbit so please no.

Everything is defined as protobuf interfaces, which is a standard used by Google and everyone else now that gRPC is so dominant. So the idea is, define the API in protobuf, code generate and implement the handlers for it. The service can be called by other services on the platform using that code generation and then an API Gateway, which Micro provides can be used to call services externally using the same format but using HTTP/JSON.

To take that even further, M3O (m3o.com) codifies protobuf to openapi specs and then generates client libraries on top. You can see some of that in https://github.com/m3o/m3o.