bdfr-html VS Pushshift-Importer

I use this with bdfr. https://github.com/BlipRanger/bdfr-html

I'm thinking i will have to get a project like redarc or BDFR-to-HTML or much more likely Pushshift-Importer which allows you to import pushshift downloads into a SQLite database. From there i would have to hook up the database to a reddit-like frontend.

You may be able to achieve that with bdfr and bdfr-html.

Now I don't know if you can find better, but you can use this alongside with this for most of reddit hoarding, see if it helps.

I have not used it myself but this might let you view the bdfr output as a website: https://github.com/BlipRanger/bdfr-html

Check out my (really beta) project to make viewing a bit easier - https://github.com/BlipRanger/bdfr-html

I'm thinking i will have to get a project like redarc or BDFR-to-HTML or much more likely Pushshift-Importer which allows you to import pushshift downloads into a SQLite database. From there i would have to hook up the database to a reddit-like frontend.

Just download the dumps from pushshift and then use Pushshift-Importer.

I wrote my own rust based importer. Feel free to use types and such from that as well.

I built a command line tool to import the dumps into sqlite if you want to give it a go. https://github.com/Paul-E/Pushshift-Importer

I wrote some code to do just this. Input the locations of the comments and submissions and it will produce an output sqlite file.

I created a pushshift importer for comments. You can find it here. It will import the comments into a sqlite database. It is written in rust and is very fast compared to python. It can import everything overnight if you have an SSD.

If you stick with SQLite, you could try creating your own sequencer. Funnel all your writes into one thread on one process, and have that thread do the writing. That way there is only ever one possible writer on the DB at a time. Here is an example what I did when I built a tool to import comments from pushshift into SQLite. When I do this on an NVME drive and I am CPU bound on decompression and JSON parsing, so the DB isn't even a bottleneck.