Reactive Streams VS pidove

Taken from https://www.reactive-streams.org/

Reactive programming is a paradigm that focuses on non-blocking and asynchronous processing of tasks. One set of specifications/abstractions for reactive programming on JVM is called Reactive Streams. Project Reactor is a message-driven, type-safe and functional implementation of Reactive Streams, and it is used by Spring (via spring-webflux module) to enable reactive web applications. Reactive streams model the data processing as a stream with one end producing the values and one end consuming them.

The reactive streams API provides the specification for non-blocking async streams processing with back pressure mechanism, and Project Reactor is an implementation written in java.

Cats Effect is a little more than just an IO effect implementation as they also provide an interface (or a standard) against implemented as typeclasses. You could think of it as a Java's Reactive Streams library which allows switching underlying implementation of actual effectful streaming.

NOTE: The Reactive Streams API and the implementation of it as provided by Spring WebFlux/Project Reactor is beyond the scope of this particular article. Please consult the appropriate documentation at the 'Web on Reactive Stack' Spring documentation site, any of several sessions I've delivered available on my YouTube channel, or by visiting the Reactive Streams and Project Reactor sites.

There is a very big design space for "Stream" APIs.

Microsoft's LINQ for instance can compile a stream operation into a SQL statement and JooQ does the same. That system offers query optimization and efficient joins that depend on the query system having complete visibility into the queries. indexes built ahead of time, etc.

Another extreme is a system like

https://www.reactive-streams.org/

that are especially good for apply a filter and map and other operations to a stream of real time events, e.g. instead of having a pull operation such as a for-loop over an Iterable, items go into the system from a stream.

I've worked on systems that use the later kind of streaming to run batch jobs and you can get great performance (780% speedup with 8 cpus) on crazy heterogenous workloads. You do have to be careful though to shut the system down or flush it out or otherwise you get wrong answers. Frequently those frameworks don't shut themselves down properly unless you implement clean shutdown yourself.

The point is that operators like "filter" and "map" and the rest are so powerful because they are portable between the minimal pidove up to a Hadoop cluster.

Backpressure is a protocol defining how to send a logical signal UP the stream with information about the downstream consumers readiness to receive more data. That backpressure signal follows the same path as data moving downstream, but in reverse. If downstream is not ready to consume, then upstream does does not send.

Reactive libraries like reactor are build on the Reactive Streams specification, just read that first sentence.

Not a well written article. "Fiber" was dropped by spec team way back in favor of "virtual thread". Mentions "Rx Java" but not http://www.reactive-streams.org/ as a standard for existing async IO. I mean anyone who has done reactive java long enough can tell you about various implementations! I expected a better article from infoworld.com

Streams is too complex for what it does and it doesn’t even parallelize well. Here is something that does roughly the same thing but I think is way better

See https://github.com/paulhoule/pidove

https://central.sonatype.com/artifact/com.ontology2/pidove

(1) It's a bit of a bad smell (which he points out) that records aren't being used much at all in the Java stdlib, I wrote something that built out stubs for the 17 and 18 stdlibs and that stood out like a sore thumb. I do like using records though.

(2) I've looked at other ways to extend the collections API and related things, see

https://github.com/paulhoule/pidove

and I think the sequenced collections could have been done better.

(3) Virtual Threads are kinda cool but overrated. Real Threads in Java are already one of the wonders of the web and perform really well for most applications. The cases where Virtual Threads are really a win will be unusual but probably important for somebody. It's a good thing it sticks to the threads API as well as it did because I know in the next five years I'm going to find some case where somebody used Virtual Threads because they thought it was cool and I'll have to switch to Real Threads but won't have a hard time doing so.

I think that guy just made up a generator class for fun. It’s not too different from the integrator except it doesn’t have a hasNext() method so it either returns results forever or it has to return a sentinel value like null or return an exception to end iteration.

Somebody could make the case that returning a sentinel value or an exception is a better API since there is no risk somebody else is going to call the next() method after you call hasNext() and next(). Writing a generator that wraps a generator is a little simpler than writing an interest or that wraps an iteration because you don’t have to write a hasNext() function, which can occasionally be awkward.

That generator library has a few functions, like map that work on generators, unfortunately the Java stdlib doesn’t come with anything like that. (There is the streams API but it is over-complicated.)

I’ll point out this library I wrote

https://github.com/paulhoule/pidove

which does a lot of what the Steams library does but it works on iterators without creating streams. If you like those generator examples you might like pidove.

As for Python it is kinda accidental that generators would up related to coroutines, that is, generators were an easy way to implement coroutines, later async/await and stuff like that got added.

I find this alien to my point of view. On the other hand, my side projects aren't driven by FOMO but are more like the "special interests" of autistic people.

Most of the time I have three side projects going on, maybe two of which are really getting the attention they deserve and one that is languishing. (See my profile to see about my current three.) Occasionally I get inspired to spend 1-4 weekends on some sudden inspiration, of which

https://github.com/paulhoule/pidove

came to completion but

https://github.com/paulhoule/ferocity

probably won't. The project I'm working the hardest on now is something that I was baffled that it didn't exist 18 years ago but felt compelled to do something out because of the Twitterinsanity last December and it turned out the technological conditions right now make it the perfect time to work on.

When I saw sequenced collections earlier I didn’t like the design but I completely approve of the latest revision. One nice thing about the process they use to develop Java is that they really do work and rework new features to make them great.

I just wish that instead of Streams they’d made something more like

https://github.com/paulhoule/pidove

... or you can just use a sane FP library like

https://github.com/paulhoule/pidove

Some people don't like the Lispy signatures so I did start coding up a version with with a fluent interface but didn't quite finish.

Inner classes are pretty useful.

This library contains a huge number of Iterables, each of which has at least one Iterator implementation.

https://github.com/paulhoule/pidove

It is convenient to let the Iterator be immutable and the Iterator be an inner class that gets its configuration information out of the Iterable.

(That said, if people really thought seriously about Iterator being a Supplier people might think more rationally about error handling. Also in a slightly parallel universe the Iterator would only have one method since remove() hardly ever gets used and having both hasNext() and next() methods is asking for bugs.)

If you were programming in Python or many other languages you could download documentation locally.

In both Python and Java doing a mini-project I frequently challenge myself to only use the standard library. It's good for practicing HackerRank-rank style programming (the fun of single-file Java)

When I am waiting for builds I sometimes hack on this

https://github.com/paulhoule/pidove

because I don't really like the Streams API and want to perfect my mastery of generics and internal DSLs.

Now that I think of it, standard-library only for node seems like a good challenge for me because I code a lot of front-end Javascript but just barely know the node API.