epanet-js VS s4

At the start up I work at [0], we use an open source library I developed to run hydraulic models of water networks in JavaScript [1].

A hydraulic model may be between 1-10MB and the simulation results can end up being 100+MB of time series data.

Other vendors with proprietary engines have to scale up servers to run their simulation engineers and will store and serve up results from a database.

Having everything done locally means we only have to store a static file and offload the simulation to the client.

Because we've architected it this way our hosting costs are low and users generally have faster access to results (assuming they're running a moderately decent machine)

[0] https://qatium.com/

[1] https://github.com/modelcreate/epanet-js

I'm a civil engineer and I wrote an open source library that compiled a C library to javascript for my own personal projects - epanet-js [1]

A water utility in Spain spun off a start up called Qatium [2] and they used my library as the engine of their simulations and asked me to join.

[1] https://github.com/modelcreate/epanet-js

[2] https://qatium.com/

I created a handful of application around water engineering/modelling [1], plus an open source library to run the simulations in javascript [2].

A water utility in Spain spun off a start up to create a similar web based water modelling application and they used my open source library.

They approached me and I joined them and have been able to maintain the open source library as part of my role.

[1] https://github.com/modelcreate/epanet-js#featured-apps

I work as a water engineer, specializing in building hydraulic models so water utilities can simulate their network.

A big part of that is calibrating them which can be time consuming, you look through hundreds of options. I create a few web based apps to help grind through these tasks but ultimately they were for my own use as a consultant to close projects quickly.

I did pull out the engine as its own open source library for other to use, and that ended up helping me get my current role where I can now maintain it and be paid at the same time.

https://github.com/modelcreate/epanet-js

https://github.com/modelcreate/epanet-js

I've built a few open source apps and few other little projects to help automate my workflow.

There are only a handful of providers of modelling software, most are commercial and one recently sold to Autodesk for $1B.

Not sure I'll convince the industry to change but I'm enjoying tinkering around and making my own small difference.

the idea of map reduce remains a good one.

there are a number of interesting innovations in streaming systems that followed, mostly around reducing latency, reducing batch size, and alternate failure/retry strategies.

even hadoop could be hard to debug when hitting a performance ceiling for challenging workloads. the streaming systems took this even further, spark being notorious for fiddle with knobs and pray the next job doesn’t fail after a few hours, again.

i played around with the thinnest possible map reduce stack a while back[1][2]. i wanted to understand the performance ceiling for different workloads without all the impenetrable layers of data bureaucracy. turns out modern network and cpu are really fast when you stop adding random software layers like lasagna.

i think the future of data, for serious workloads, is gonna be bespoke. the primitives are just too good now.

1. https://github.com/nathants/s4

2. https://github.com/nathants/bsv

pipes are great. is the other process on another cpu or another machine? honestly who cares.

https://github.com/nathants/s4/blob/master/examples/nyc_taxi...

i dragged my feet on go for a long time. i also thought that skipping go and moving to rust was the play. a few years later, i still write python often, but i don’t build systems with it. python i now use like bash, to glue things together and automate random things. it’s a fantastic language and i will never drop it.

the verbosity of go is the biggest hurdle for a pythonista. the thought of giving up context managers, decorators, iterators, comprehensions, exceptions, coroutines, it’s unthinkable. in comparison go is ugly. your aesthetic mind screams in protest.

write go full time. dive in. as months pass, not only will those aesthetic objections fade, your mental model from python cleanly transforms to go. go is what mypy tried to be. the cost was aesthetic changes. the benefit is worth it.

the zen of python says if it’s easy to explain it might be a good idea. this is go, and it is.

i rebuilt a reasonably sized project from python[1] to go[2] over the last few years. i also have a system that i maintained both python[3] and go[4] implementations for, sharing a test suite in python.

go, like python, is fantastic. use both in whatever amount works for you. don’t read about them, build with them. you won’t regret it.

1. https://github.com/nathants/cli-aws/tree/bb78e529e7d1d3f95ac...

2. https://github.com/nathants/libaws

3. https://github.com/nathants/s4/tree/python

4. https://github.com/nathants/s4