zipline VS bcolz

https://github.com/quantopian/zipline (event-driven)

Zipline by Quantopian: An Open-Source tool for algorithmic trading. It is a platform for developing and testing quantitative trading strategies using Python.

For personal use only. I'm currently looking at QuantConnect's LEAN and Quantopian's Zipline (which hasn't seen any updates since 2020, presumably because Quantopian was dissolved).

(1) I remember there were decent libraries for this setting a while back. Maybe take a look at Quantopian/Zipline.

zipline – Zipline is a Pythonic algorithmic trading library. It is an event-driven system that supports both backtesting and live trading.

I found two opensource tools 1. .https://github.com/quantopian/zipline Quantopian 2. https://analyzingalpha.com/backtrader-backtesting-trading-strategies backtrader

Some of the notable libraries in Python are backtesting.py, bt and zipline. Personally I like bt the most, as its tree model makes the most intuitive sense.

What you need for your use case is a column-oriented store. I recommend explore bcolz or apache arrow for a column file-based systems. These are very fast, support memory mapping, uses compression and SSD speed (and even CPU architecture, in case of arrow) optimally almost out of the box, and has good interfaces to Numpy and Pandas (in case you are using Python for final data consumption and analysis). The columnar structure makes it easy to add or delete a column easily (or even dynamically). If you need a more scalable (albeit at the cost of speed) solution, you can devise a schema over a regular columnar db or an nosql db - see arctic from Man group for an example.