snowflake VS cockroach

There are so many of them in big data, e.g. Kafka, Spark, Flink, Delta, Snowplow, Finagle, Deequ, CMAK, OpenWhisk, Snowflake, TheHive, TVM-VTA, etc.

Twitter built Snowflake https://github.com/twitter-archive/snowflake/tree/snowflake-2010

I get similar connection-security messages when attempting to connect to the package archive at https://elpa.nongnu.org/ —again, in both *Package* and in *eww*—while elpa.gnu.org seems to be just flat out nonresponsive. However, other https sites including https://duckduckgo.com/about and https://twitter.com work just fine.

Twitter started out with a huge focus on their API. Developers could get almost any data from Twitter they wanted - trends, hashtags, user stats - and they built some really cool stuff with it. This massive amount of open data and the tools people built actually attracted more users to Twitter. Companies could easily hook into the Twitter API to let users share their content on Twitter without leaving their site, and Twitter in turn got even more content on the platform.

Developed and Maintained By – Twitter

PS C:\Users\Admin\Downloads> .\yt-dlp.exe "https://twitter.com/i/broadcasts/1dRKZlzzbMbJB" --verbose [debug] Command-line config: ['https://twitter.com/i/broadcasts/1dRKZlzzbMbJB', '--verbose'] [debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252 [debug] yt-dlp version 2021.09.25 (exe) [debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0 [debug] exe versions: none [debug] Optional libraries: Crypto, mutagen, sqlite, websockets [debug] Proxy map: {} [debug] [twitter:broadcast] Extracting URL: https://twitter.com/i/broadcasts/1dRKZlzzbMbJB [twitter:broadcast] 1dRKZlzzbMbJB: Downloading guest token [twitter:broadcast] 1dRKZlzzbMbJB: Downloading JSON metadata [twitter:broadcast] 28_1445749360700977153: Downloading JSON metadata [twitter:broadcast] 1dRKZlzzbMbJB: Downloading m3u8 information [debug] Default format spec: best/bestvideo+bestaudio [info] 1dRKZlzzbMbJB: Downloading 1 format(s): replay-2750 [debug] Invoking downloader on "https://prod-fastly-us-east-1.video.pscp.tv/Transcoding/v1/hls/jN6CaxrA9lpyuGd0L_kWYPCorYH27HnjKV70nxcVASsjFnGajs0gDWIsABSOYQJd-E7xK7kFJv34xb38IZi6cA/transcode/us-east-1/periscope-replay-direct-prod-us-east-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJFbmNvZGVyU2V0dGluZyI6ImVuY29kZXJfc2V0dGluZ183MjBwMzBfMTAiLCJIZWlnaHQiOjcyMCwiS2JwcyI6Mjc1MCwiV2lkdGgiOjEyODB9.ldktM4fCFRfkP4ZEBfZPKtlAUNAcTPkoz994YJAzWpE/tw_playlist_16813172078088237633.m3u8?type=replay" [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 2409 [download] Destination: Drone Racing League - 2021-22 DRL World Championship _ Twin Cities [1dRKZlzzbMbJB].mp4 [download] Drone Racing League - 2021-22 DRL World Championship _ Twin Cities [1dRKZlzzbMbJB].mp4.part-Frag1 has already been downloaded [download] 0.0% of ~1.09GiB at Unknown speed ETA Unknown ETAERROR: unable to download video data: HTTP Error 400: Bad Request Traceback (most recent call last): File "yt_dlp\YoutubeDL.py", line 2829, in process_info File "yt_dlp\YoutubeDL.py", line 2489, in dl File "yt_dlp\downloader\common.py", line 408, in download File "yt_dlp\downloader\hls.py", line 344, in real_download File "yt_dlp\downloader\fragment.py", line 478, in download_and_append_fragments File "yt_dlp\downloader\fragment.py", line 352, in decrypt_fragment File "yt_dlp\downloader\fragment.py", line 344, in _get_key File "yt_dlp\YoutubeDL.py", line 3256, in urlopen File "urllib\request.py", line 531, in open File "urllib\request.py", line 640, in http_response File "urllib\request.py", line 569, in error File "urllib\request.py", line 502, in _call_chain File "urllib\request.py", line 649, in http_error_default urllib.error.HTTPError: HTTP Error 400: Bad Request

a[href^="https://twitter.com/"] is not part of #yle-consent-sdk-container children

yle.fi###yle-consent-sdk-container:style(visibility: hidden) yle.fi###yle-consent-sdk-container:has(~ .yle__app a[href^="https://twitter.com/"]):style(visibility: visible !important)

If you run that and then look in your Network tab again, you'll see that the response includes a header that says Access-Control-Allow-Origin: *. The * means that this image is allowed to be requested from any origin whatsoever- this is presumably because the NYT want this image to show up in social media shares. This could also be restricted- for example, if for some reason they only wanted the image to be allowed to be requested from Twitter, they could send a header that said Access-Control-Allow-Origin: https://twitter.com.

// ==UserScript== // @name Twitter to nitter // @namespace null // @version 0.1 // @description Redirect twitter to nitter // @match https://twitter.com/* // @run-at document-start // ==/UserScript== redirectToPage("https://twitter.com", "https://nitter.net"); function redirectToPage(page1, page2) { if(window.location.href.indexOf(page1) != -1) { window.location.href = page2 + window.location.pathname; } }

Cockroachdb created a go runtime patch which measures the Grunning time of a goroutine: https://github.com/cockroachdb/cockroach/pull/82356. It doesn't entirely solve the problem though.

Our entire backend is written in Go. We've built a platform that allows other companies to offer automatic data syncing to their customers' data warehouses. Go works great for building distributed systems like this (see K8s). We're not the only ones in the space building data intensive applications with Go. Pachyderm, Pinecone, Cockroach Labs and are all also doing it. We've been quite happy with how Go has worked for us.

I would recommend researching CockroachDB. It does what you're describing natively. The basis of it's design is to distribute data based on geography. It's inexpensive and very simple to install and manage. It's based on Postgres ( but doesn't support all Postgres features). We've been using it for about 1 year and it meets our needs for some of our workload. www.cockroachlabs.com

CockroachDB

You absolutely can write very high performance software in Go, that's kind of the point. You can efficiently interface with C libraries. You can create the sort of software everyone says should be done in Rust, like databases and web servers and system orchestration and games and every other goddamn thing that people will say isn't the right choice for Go.

PostgreSQL Parser separated from CockroachDB, a distributed DB.

Spencer Kimball (now CEO at CockroachDB) wrote an interesting article on this topic in 2021 where they created spencerkimball/stargazers based on a Python script. So I started thinking: could I create a data pipeline using Nifi and Kafka (two OSS tools often used with Druid) to get the API data into Druid - and then use SQL to do the analytics? The answer was yes! And I have documented the outcome below. Here’s my analytical pipeline for Github stars data using Nifi, Kafka and Druid.