reddit_sse_stream
dpark
| reddit_sse_stream | dpark | |
|---|---|---|
| 6 | - | |
| 47 | 2,691 | |
| - | 0.0% | |
| 0.0 | 0.0 | |
| almost 2 years ago | over 3 years ago | |
| Python | Python | |
| MIT License | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
reddit_sse_stream
-
Pushshift Live Again and How Moderators Can Request Pushshift Access
Will you still be providing the SSE Stream API using this new bearer token authentication?
-
Thoughts on a pushshift alternative
Pushshift used to have this https://github.com/pushshift/reddit_sse_stream
-
Introducing Sunbelt, a new service similar to Pushshift
while the point of pushshift is absolute collection of data, it also had the https://github.com/pushshift/reddit_sse_stream for getting notified of new posts, which was a godsend for bots (moderation and i guess otherwise) during the short time it worked correctly
-
https://np.reddit.com/r/rust/comments/lxkylx/how_to_use_async_sse_to_read_from_a_remote_sse/gpzq5wn/
use std::io::{Error, ErrorKind}; use futures::prelude::*; #[tokio::main] async fn main() { let url = "http://stream.pushshift.io"; let stream = reqwest::get(url) .await .unwrap() .bytes_stream(); let mut reader = async_sse::decode( stream .map_err(|e| Error::new(ErrorKind::Other, e)) .into_async_read() ); let event = reader.next().await.unwrap(); println!("{:?}", event); }
-
How to use async_sse to read from a remote SSE stream?
async fn main() { let url = "http://stream.pushshift.io";
dpark
We haven't tracked posts mentioning dpark yet.
Tracking mentions began in Dec 2020.
What are some alternatives?
Apache Spark - Apache Spark - A unified analytics engine for large-scale data processing
mrjob - Run MapReduce jobs on Hadoop or Amazon Web Services
streamparse - Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
dumbo - Python module that allows one to easily write and run Hadoop programs.
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
tdigest - t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
data-science-ipython-notebooks - Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.