py-caskdb VS fio

self plug: I wanted to learn how databases work internally, like how they store and retrieve data, build indexes, etc.

I built an educational KV store to teach someone to write a database from scratch. I have set up this project in TDD fashion with the tests. So, you start with simple functions, pass the tests, and the difficulty level goes up. There are hints if you get stuck. When all the tests pass, you would have written a persistent key-value store in the end.

link: https://github.com/avinassh/py-caskdb

link: https://github.com/avinassh/py-caskdb

If you like databases, you start exploring the internals and start writing one! There is no going back once you dig deep into storage internals, KV stores, and distributed systems.

plug: I made an educational project which can help you write a database in python, from scratch - https://github.com/avinassh/py-caskdb

Build your own disk based KV store

link - https://github.com/avinassh/py-caskdb

Assuming two systems use flash storage, network bandwidth is identical and it is configured the same way, there should be an issue within the PC, either system or storage drive. Check the system logs for errors and warning events related to data transfer from/to NAS. Try to benchmark the PCs' disks using fio to confirm they have similar performance. https://github.com/axboe/fio

Not specifically addressing your question, but when you get to the point of wanting to start doing some experiments you may find that 'fio' [1] is very handy.

[1] https://github.com/axboe/fio

The dd is not a good benchmarking tool, you should use something like fio and probably tune it to use the ioengine most similar to your use case (eg. a database server will probably use some async IO interface). In your first example (with bs=1G) probably something (the guest OS, the qemu/kvm or the host OS) have split into smaller chunks anyway.

All linux tests are run with fio 3.32 (github) with future commit 03900b0bf8af625bb43b10f0627b3c5947c3ff79 manually applied.

FIO: https://github.com/axboe/fio

Agree, I used flex/yacc to add an arithmetic expression evaluator to fio a few years back to allow simple math with some units in fio's job files, and for stuff like that, they're fine, but I wouldn't want to use them for a real language, the error handling is kind of a nightmare.

Flexible I/O Tester (fio-3.33): https://github.com/axboe/fio