reddit_sse_stream VS vaex

Will you still be providing the SSE Stream API using this new bearer token authentication?

Pushshift used to have this https://github.com/pushshift/reddit_sse_stream

while the point of pushshift is absolute collection of data, it also had the https://github.com/pushshift/reddit_sse_stream for getting notified of new posts, which was a godsend for bots (moderation and i guess otherwise) during the short time it worked correctly

use std::io::{Error, ErrorKind}; use futures::prelude::*; #[tokio::main] async fn main() { let url = "http://stream.pushshift.io"; let stream = reqwest::get(url) .await .unwrap() .bytes_stream(); let mut reader = async_sse::decode( stream .map_err(|e| Error::new(ErrorKind::Other, e)) .into_async_read() ); let event = reader.next().await.unwrap(); println!("{:?}", event); }

async fn main() { let url = "http://stream.pushshift.io";

Try vaex, vaex, using lazy evaluation and parallel calculations, you should be fine.

I'd recommend QuestDB. Worked with it multiple times for different algorithmic trading needs and it didn't disappoint. If you want to load data fast, I'd recommend this Python library.

How about vaex?

You might find vaex (https://github.com/vaexio/vaex) interesting if you work with HDF5.

My guess is that should be possible, feel free to hop onto https://github.com/vaexio/vaex/discussions !