Pushshift API VS PRAW

I use https://camas.unddit.com all the time, and the full pushshift API for more complicated searches. It's incredibly useful to me

Directly querying it takes a bit of technical knowledge however people have put together websites that provide an easy way of doing so, at the expense of having slightly less features

Don’t forget PushShift[1] for getting Reddit data!

It's quite feasible to use Reddit's API to fetch every comment as it's posted. There's also an API called Pushshift that constantly scrapes Reddit and lets you query for subsets of the data, such as "the most recent comments containing a particular keyword".

This exists: https://github.com/pushshift/api

Hi, I am trying to pull a large amount of data via the PullShift reddit API (10's of millions of records) and I was wondering how an API rate limiter works? I am wondering this because, as of now, it will take A LONG time for me to pull all the data I need, even using something such as PMAW. So I am looking for solutions to speed up the process, mainly using multiple machines in parallel to pull different subsets of the data.

To get the relevant IDs, you can use something like PRAW to query the subreddit for the top 1000 posts for example.

Huge thanks to the contributors to PRAW, which is the Python package that does all the heavy lifting relating to Reddit's API that I need for this script.

However, in practice duplicate items were yielded with these smaller numbers. So I increased the limit briefly to 250 in October 2016, and then increased it finally to 301 in December 2016 in order to resolve https://github.com/praw-dev/praw/issues/673. That issue provides an explanation for how 301 came to be.

I have a pretty specific question about PRAW about the code shown here in the source repository. Perhaps this is best directed at one of the authors like /u/bboe.

Woah, dude, you're doing a lot of the heavy lifting trying to make sense of reddit's API yourself. Why not use an existing API wrapper for reddit such as praw? That way, authenticate() isn't really needed and main() could be shortened substantially.

the reddit API is actually really nice: https://github.com/praw-dev/praw

Ha, nearly a year ago now I started implementing something similar in Python using the excellent PRAW, but it never got finished.

That makes sense. For the second one you might take some design tips from praw: https://github.com/praw-dev/praw/tree/master/praw

praw is sequential and he is looking to leverage asyncpraw to work on analytics like comments, submissions, votes etc., any suggestions on this front would help as well.