snowflake VS Pushshift API

There are so many of them in big data, e.g. Kafka, Spark, Flink, Delta, Snowplow, Finagle, Deequ, CMAK, OpenWhisk, Snowflake, TheHive, TVM-VTA, etc.

Twitter built Snowflake https://github.com/twitter-archive/snowflake/tree/snowflake-2010

I get similar connection-security messages when attempting to connect to the package archive at https://elpa.nongnu.org/ —again, in both *Package* and in *eww*—while elpa.gnu.org seems to be just flat out nonresponsive. However, other https sites including https://duckduckgo.com/about and https://twitter.com work just fine.

Twitter started out with a huge focus on their API. Developers could get almost any data from Twitter they wanted - trends, hashtags, user stats - and they built some really cool stuff with it. This massive amount of open data and the tools people built actually attracted more users to Twitter. Companies could easily hook into the Twitter API to let users share their content on Twitter without leaving their site, and Twitter in turn got even more content on the platform.

Developed and Maintained By – Twitter

PS C:\Users\Admin\Downloads> .\yt-dlp.exe "https://twitter.com/i/broadcasts/1dRKZlzzbMbJB" --verbose [debug] Command-line config: ['https://twitter.com/i/broadcasts/1dRKZlzzbMbJB', '--verbose'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] yt-dlp version 2021.09.25 (exe) [debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0 [debug] exe versions: none [debug] Optional libraries: Crypto, mutagen, sqlite, websockets [debug] Proxy map: {} [debug] [twitter:broadcast] Extracting URL: https://twitter.com/i/broadcasts/1dRKZlzzbMbJB [twitter:broadcast] 1dRKZlzzbMbJB: Downloading guest token [twitter:broadcast] 1dRKZlzzbMbJB: Downloading JSON metadata [twitter:broadcast] 28_1445749360700977153: Downloading JSON metadata [twitter:broadcast] 1dRKZlzzbMbJB: Downloading m3u8 information [debug] Default format spec: best/bestvideo+bestaudio [info] 1dRKZlzzbMbJB: Downloading 1 format(s): replay-2750 [debug] Invoking downloader on "https://prod-fastly-us-east-1.video.pscp.tv/Transcoding/v1/hls/jN6CaxrA9lpyuGd0L_kWYPCorYH27HnjKV70nxcVASsjFnGajs0gDWIsABSOYQJd-E7xK7kFJv34xb38IZi6cA/transcode/us-east-1/periscope-replay-direct-prod-us-east-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJFbmNvZGVyU2V0dGluZyI6ImVuY29kZXJfc2V0dGluZ183MjBwMzBfMTAiLCJIZWlnaHQiOjcyMCwiS2JwcyI6Mjc1MCwiV2lkdGgiOjEyODB9.ldktM4fCFRfkP4ZEBfZPKtlAUNAcTPkoz994YJAzWpE/tw_playlist_16813172078088237633.m3u8?type=replay" [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 2409 [download] Destination: Drone Racing League - 2021-22 DRL World Championship _ Twin Cities [1dRKZlzzbMbJB].mp4 [download] Drone Racing League - 2021-22 DRL World Championship _ Twin Cities [1dRKZlzzbMbJB].mp4.part-Frag1 has already been downloaded [download] 0.0% of ~1.09GiB at Unknown speed ETA Unknown ETAERROR: unable to download video data: HTTP Error 400: Bad Request Traceback (most recent call last): File "yt_dlp\YoutubeDL.py", line 2829, in process_info File "yt_dlp\YoutubeDL.py", line 2489, in dl File "yt_dlp\downloader\common.py", line 408, in download File "yt_dlp\downloader\hls.py", line 344, in real_download File "yt_dlp\downloader\fragment.py", line 478, in download_and_append_fragments File "yt_dlp\downloader\fragment.py", line 352, in decrypt_fragment File "yt_dlp\downloader\fragment.py", line 344, in _get_key File "yt_dlp\YoutubeDL.py", line 3256, in urlopen File "urllib\request.py", line 531, in open File "urllib\request.py", line 640, in http_response File "urllib\request.py", line 569, in error File "urllib\request.py", line 502, in _call_chain File "urllib\request.py", line 649, in http_error_default urllib.error.HTTPError: HTTP Error 400: Bad Request

a[href^="https://twitter.com/"] is not part of #yle-consent-sdk-container children

yle.fi###yle-consent-sdk-container:style(visibility: hidden) yle.fi###yle-consent-sdk-container:has(~ .yle__app a[href^="https://twitter.com/"]):style(visibility: visible !important)

If you run that and then look in your Network tab again, you'll see that the response includes a header that says Access-Control-Allow-Origin: *. The * means that this image is allowed to be requested from any origin whatsoever- this is presumably because the NYT want this image to show up in social media shares. This could also be restricted- for example, if for some reason they only wanted the image to be allowed to be requested from Twitter, they could send a header that said Access-Control-Allow-Origin: https://twitter.com.

// ==UserScript== // @name Twitter to nitter // @namespace null // @version 0.1 // @description Redirect twitter to nitter // @match https://twitter.com/* // @run-at document-start // ==/UserScript== redirectToPage("https://twitter.com", "https://nitter.net"); function redirectToPage(page1, page2) { if(window.location.href.indexOf(page1) != -1) { window.location.href = page2 + window.location.pathname; } }

I use https://camas.unddit.com all the time, and the full pushshift API for more complicated searches. It's incredibly useful to me

Directly querying it takes a bit of technical knowledge however people have put together websites that provide an easy way of doing so, at the expense of having slightly less features

Don’t forget PushShift[1] for getting Reddit data!

It's quite feasible to use Reddit's API to fetch every comment as it's posted. There's also an API called Pushshift that constantly scrapes Reddit and lets you query for subsets of the data, such as "the most recent comments containing a particular keyword".

This exists: https://github.com/pushshift/api

Hi, I am trying to pull a large amount of data via the PullShift reddit API (10's of millions of records) and I was wondering how an API rate limiter works? I am wondering this because, as of now, it will take A LONG time for me to pull all the data I need, even using something such as PMAW. So I am looking for solutions to speed up the process, mainly using multiple machines in parallel to pull different subsets of the data.