Globals or not globals?

• v2-tooling

  6 3 7.9 Shell

  Tools for handling v2.*.* syntax

  

Recently, I've implemented my first parser in Bash. It works, but the problem is how slow it works on thousands of files. One of the issues that almost all functions accepts some page as an input and produce some output, which means that they reparse the same page too many times. I don't use global variables now to cache parsed results to use them later. Its speed is not a big issue when just using it for small amount of files.

• forkrun

  18 136 9.7 Shell

  runs multiple inputs through a script/function in parallel using bash coprocs

However, if you have bash 4 or later, using bash coprocs can offer much better performance. The idea is that you spawn a bunch of persistent coprocs and pipe data to them, allowing you to parallelize the loop without forking each individual iteration. I wrote / am writing a utility called forkrun that parallelizes loops using bash coprocs. You're welcome to use it, though heads up it is still in beta (almost everything works, but I am still debugging the -k flag sometimes not working properly...-k ensures the output order is the same as input order). If nothing else it gives a real world example of parallelizing using coprocs.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project