BeanstalkD
nifi
Our great sponsors
| BeanstalkD | nifi | |
|---|---|---|
| 14 | 35 | |
| 6,472 | 4,381 | |
| 0.4% | 3.1% | |
| 0.0 | 9.9 | |
| 5 months ago | 4 days ago | |
| C | Java | |
| GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
BeanstalkD
-
Ruby 3.3
There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.
https://beanstalkd.github.io/
-
A Developer's Journal: Simplifying the Twelve-Factor App
Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)
- Load Balancing
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.
If you're measuring throughput in jobs/s, use a real work queue.
-
Christmas giveaway: 10 copies of my book Domain-driven Design with Golang book, also AMA
Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.
-
PHP parallel processing idea
Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.
- How to do distributed cronjobs with worker queues?
-
Write Your Own Task Queue
The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:
https://beanstalkd.github.io/
- Golang task queue
-
What are some popular background job processing frameworks in the Rust ecosystem?
It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.
nifi
- FLaNK Stack Weekly 19 Feb 2024
- Ask HN: What are some unpopular technologies you wish people knew more about?
- FLaNK Stack Weekly for 13 November 2023
-
Ask HN: What low code platforms are worth using?
Apache NIFI (https://nifi.apache.org/).
It uses the concept of Flow-based programming. Also its so underacknolged but this tool is very flexible. I have used as an Event Bus all the 3rd-Party Integrations.
- Apache Nifi: easy to use, powerful, reliable system to process, distribute data
- Tool decision - What architecture would you choose and why?
-
Help with choosing techstack for a new DE team
Presently setting up Apache Nifi + Apache MiNiFi for the ETL portion of my work. NiFi was easy enough to figure out; but the docs for MiNiFi have been a pain due to differences between the Java and C++ versions. I then entirely configured it with the Java version so that it was easier to search for answers for the MiNiFi yaml syntax.
-
MS SQL Change Data Capture
Found it
-
Is there something like airflow but written in Scala/Java?
Apache Camel Apache Nifi Spring Cloud
-
Json splitting and Rerouting (new to nifi)
NIFI, like most Apache projects does most of its discussion on its mailing lists, but also has a slack.
What are some alternatives?
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
Logstash - Logstash - transport and process your logs, events, or other data
Apache Kafka - Mirror of Apache Kafka
superset - Apache Superset is a Data Visualization and Data Exploration Platform
Gearman
meltano
NATS - High-Performance server for NATS.io, the cloud and edge native messaging system.
meltano - Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
celery - Distributed Task Queue (development branch)
Pandas - Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
nsq - A realtime distributed messaging platform
Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum: