Logstash
nifi
Our great sponsors
| Logstash | nifi | |
|---|---|---|
| 5 | 35 | |
| 13,984 | 4,369 | |
| 0.5% | 2.9% | |
| 9.6 | 9.9 | |
| 4 days ago | 3 days ago | |
| Java | Java | |
| GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Logstash
-
Simple way to centralize my server logs?
There are probably too many to chose from. Logstash, Promtail, Vector, Filebeat, FluentD, Logagent and probably many more
-
Looking for open source projects that use data pipelines and big data flows
Is logstash the kind of project you are looking for? https://github.com/elastic/logstash
nifi
- FLaNK Stack Weekly 19 Feb 2024
- Ask HN: What are some unpopular technologies you wish people knew more about?
- FLaNK Stack Weekly for 13 November 2023
- Tool decision - What architecture would you choose and why?
-
Is there something like airflow but written in Scala/Java?
Apache Camel Apache Nifi Spring Cloud
-
Your opinion on Kong
This suggestion isn't a standard one, but when a coworker and I were looking for API gateways with a very specific feature set, we couldn't find a single one to do what we needed. We did, however, come across Apache NiFi. It's a flow-based programming tool that allowed us to translate an http-based request to streaming text sent via netcat.
-
S3 to S3 transform
For a simple sequential Pipeline, my goto would be Apache Camel. As soon as you want complexity its either Apache Nifi or a micro service architecture.
-
Analysing Github Stars - Extracting and analyzing data from Github using Apache NiFi®, Apache Kafka® and Apache Druid®
Spencer Kimball (now CEO at CockroachDB) wrote an interesting article on this topic in 2021 where they created spencerkimball/stargazers based on a Python script. So I started thinking: could I create a data pipeline using Nifi and Kafka (two OSS tools often used with Druid) to get the API data into Druid - and then use SQL to do the analytics? The answer was yes! And I have documented the outcome below. Here’s my analytical pipeline for Github stars data using Nifi, Kafka and Druid.
-
Is there any automation solution that isn't "only" CI/CD except Jenkins?
For dataflow pipelines I'm really a fan of apache nifi https://nifi.apache.org/
-
Read database each 5 sec and dispatch event
Pretty sure you can do this with a NiFi connector. https://nifi.apache.org
What are some alternatives?
Logback - The reliable, generic, fast and flexible logging framework for Java.
Apache Log4j 2 - Apache Log4j 2 is a versatile, feature-rich, efficient logging API and backend for Java.
graylog - Free and open log management
Flume - Mirror of Apache Flume
superset - Apache Superset is a Data Visualization and Data Exploration Platform
Logbook - An extensible Java library for HTTP request and response logging
SLF4J - Simple Logging Facade for Java
Fluentd - Fluentd: Unified Logging Layer (project under CNCF)
meltano
kibana - Your window into the Elastic Stack
meltano - Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Hindsight - Hindsight - light weight data processing skeleton