RedditLemmyImporter VS reddit-archivar

There is https://github.com/rileynull/RedditLemmyImporter if someone wants to help us move to /c/libgen

For https://github.com/rileynull/RedditLemmyImporter it needs a Lemmy instance, and an API key for which https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki

Actually there are a few such as RedditLemmyImporter by rileynulll.

The issue is - how? I've seen that there have been tools created to back up subreddits to Lemmy, but these rely on having access to the instance database in order to effectively "seed" it; that's not possible for me as a moderator/user.

Currently, the RedditLemmyImporter tool is the best tool for migrating your subreddits away from Reddit. I'm not sure how many moderators are willing to go through the hassle of installing and running this script to migrate their communities. But I'm here to inform you that the tool does exist. If you're facing any technical difficulties with the tool, I think r/LemmyMigration, r/DataHoarder or the Github Issues page will provide the most technical support. I do apologise for the lack of documentation on the Github page though (its not my script).

For literally migrating posts and comments over to Lemmy: https://github.com/rileynull/RedditLemmyImporter

Haven't tried it, but this comment on /r/DataHoarder mentioned these two repos:

https://github.com/rileynull/RedditLemmyImporter

https://github.com/rileynull/RedditLemmyImporter might be what you are looking for?

I was focussing mostly on cyber security related subreddits because the vulnerability and exploit discussions were of great value to me.

I built a little scraper in golang that stores the JSON data (instead of the HTML which the archive warrior stores) to save hdd storage.

I managed to discover and scrape around 80GB of reduced JSON data, but have no idea what to do with this now. I wanted to build myself a little minimalistic web interface so I can do a text/keyword search.

The problem with reddit's API is that it only shows 1000 entries over 10 pages in every api. Meaning hot/top/new, and search results are limited. If you have more links related to the keyword, you won't discover more.

So you need a very specific keyword list to be able to discover more posts.

[1] https://github.com/cookiengineer/reddit-archivar

Inspired by the ongoing call-to-action by the Internet Archive team over at /r/DataHoarder [1], I've decided I want to try to preserve all cybersecurity related subreddits. [2]

For people that don't know what's going on: There's a likelihood that the try to monetize the Reddit API will lead to a lot of moderators quitting the platform, and it could be that a lot of subreddits are going to be set on private and/or their threads are going to be deleted. At least that's kind of the fear from the ongoing moderator strike.

In my case I learned a LOT from reddits' discussions about malware, exploits and how they work, and without those I certainly wouldn't be where I am today ... so I'm trying to preserve them.

As the Archive Warrior only scrapes the HTML directly to the Web Archive, I'm trying to preserve the data itself directly as JSON files; with intent to store it later on IPFS (having been inspired a couple days ago by the-eye-team's effort to archive RARBG on IPFS).

I just wanted to let people know here about the tool, and in case you want to archive your favorite subreddits, feel free to modify it.

There are some limitations though, because listings (new/hot/top/search) are all limited to 1000 entries, which means that the discovery of old threads is quite limited.

Keyword search increases the discovery of old threads. In my case I'm searching for a lot of keywords (like CVE, RCE, vulnerability etc) in order to discover more threads.

Would love to hear feedback, currently it's just a prototypical quick n' dirty tool because the threat of my favorite subreddits going dark is quite immediate. I tried to reduce as much noise from the schema as possible, and the tool is only archiving the subreddit threads and comments, with the idea to be able to scrape the websites/blog articles at a later point in time.

[1] https://old.reddit.com/r/DataHoarder/comments/142l1i0/archiveteam_has_saved_over_108_billion_reddit/

[2] https://github.com/cookiengineer/reddit-archivar