druid-datasets VS stargazers

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Nifi is very useful when data needs to be loaded from different sources. In this case, I will nifi to access the Github API as it is very easy to make repeated calls to a Http endpoint and get data from multiple pages. You can see what I did by downloading NiFi yourself and then adding my template from the Druid Datasets repo: https://github.com/implydata/druid-datasets/blob/main/githubstars/github_stars.xml

I used tools (such at this one and this one) to see what other repos our stargazers were starring most frequently. The stargazers of these "other" commonly starred repos became an expanded pool of potential stargazers for Preevy

Spencer Kimball (now CEO at CockroachDB) wrote an interesting article on this topic in 2021 where they created spencerkimball/stargazers based on a Python script. So I started thinking: could I create a data pipeline using Nifi and Kafka (two OSS tools often used with Druid) to get the API data into Druid - and then use SQL to do the analytics? The answer was yes! And I have documented the outcome below. Here’s my analytical pipeline for Github stars data using Nifi, Kafka and Druid.

It's analytics. Google search brings up

https://www.cockroachlabs.com/blog/what-can-we-learn-from-ou...

"Years ago I dedicated a Flex Friday (our version of 20% time) to stargazers, a tool to query the CockroachDB repository for information about its GitHub stars and analyze the results. At the time of writing, we had 6,000+ stars (which felt like a lot), and the data in this blog will be based on that original set of 6,000 stargazers."

Which links to

https://github.com/spencerkimball/stargazers

"GitHub allows visitors to star a repo to bookmark it for later perusal. Stars represent a casual interest in a repo, and when enough of them accumulate, it's natural to wonder what's driving interest. Stargazers attempts to get a handle on who these users are by finding out what else they've starred, which other repositories they've contributed to, and who's following them on GitHub."

Go ahead and use the Stargazers repo yourself to analyze yours (or anyone else’s) repo’s trends. Depending upon the number of Stargazers you have, it can take some time. It took us about 7-8 hours (with a 5K/Hr rate limit).