stargazers VS nifi

I used tools (such at this one and this one) to see what other repos our stargazers were starring most frequently. The stargazers of these "other" commonly starred repos became an expanded pool of potential stargazers for Preevy

Spencer Kimball (now CEO at CockroachDB) wrote an interesting article on this topic in 2021 where they created spencerkimball/stargazers based on a Python script. So I started thinking: could I create a data pipeline using Nifi and Kafka (two OSS tools often used with Druid) to get the API data into Druid - and then use SQL to do the analytics? The answer was yes! And I have documented the outcome below. Here’s my analytical pipeline for Github stars data using Nifi, Kafka and Druid.

It's analytics. Google search brings up

https://www.cockroachlabs.com/blog/what-can-we-learn-from-ou...

"Years ago I dedicated a Flex Friday (our version of 20% time) to stargazers, a tool to query the CockroachDB repository for information about its GitHub stars and analyze the results. At the time of writing, we had 6,000+ stars (which felt like a lot), and the data in this blog will be based on that original set of 6,000 stargazers."

Which links to

https://github.com/spencerkimball/stargazers

"GitHub allows visitors to star a repo to bookmark it for later perusal. Stars represent a casual interest in a repo, and when enough of them accumulate, it's natural to wonder what's driving interest. Stargazers attempts to get a handle on who these users are by finding out what else they've starred, which other repositories they've contributed to, and who's following them on GitHub."

Go ahead and use the Stargazers repo yourself to analyze yours (or anyone else’s) repo’s trends. Depending upon the number of Stargazers you have, it can take some time. It took us about 7-8 hours (with a 5K/Hr rate limit).

Apache NIFI (https://nifi.apache.org/).

It uses the concept of Flow-based programming. Also its so underacknolged but this tool is very flexible. I have used as an Event Bus all the 3rd-Party Integrations.

Presently setting up Apache Nifi + Apache MiNiFi for the ETL portion of my work. NiFi was easy enough to figure out; but the docs for MiNiFi have been a pain due to differences between the Java and C++ versions. I then entirely configured it with the Java version so that it was easier to search for answers for the MiNiFi yaml syntax.

Found it

Apache Camel Apache Nifi Spring Cloud

NIFI, like most Apache projects does most of its discussion on its mailing lists, but also has a slack.