Concurrent Ruby VS ruby-vips

Let’s say we need to traverse through thousands of files in our S3 Storage in a Ruby app. Let’s say we have a bunch of logs there that we need to read every day and process. If we just use a straightforward approach, like opening, reading, and processing every file one by one, our solution will work, but It will take a lot of time to process. So we need to improve the speed. Here ruby-concurrent gem is our helper https://github.com/ruby-concurrency/concurrent-ruby. Why do we need this gem? Because It’s simpler to use than Threads and this gem has a lot more features in It. In this article, we will use Concurrent::Promises.future as the most common use of concurrent code. Because reading a file from S3 is an IO operation, we can get a huge benefit in speed if we gonna use concurrent code doing HTTP requests. Remember that concurrency will not give you speed improvements if in every Promise or Thread you will do any calculations. Because of Ruby GIL, every thread will be blocked until calculations are finished.

concurrent-ruby - to add async requests ability. https://github.com/ruby-concurrency/concurrent-ruby

After this, I took a look at the semaphore class in the popular library, concurrent-ruby to see how they implement it, and I learnt about something new: condition variables. And Ruby comes with this included!

https://github.com/ruby-concurrency/concurrent-ruby has great docs if someone is looking for alternatives.

I wonder how this would compare to using concurrent-ruby under ruby 2.7, especially in a real-world setting (where the calls are actually to external services that return and buffer data, instead of just sleep). The author says that he's felt that ruby threads "feel easy to mess up," but I've found that concurrent-ruby makes it pretty simple, and performant enough even with the GIL.

As I could not find a clear example about how to rescue exceptions from Concurrent::Promises (part of the Concurrent Ruby gem ) I read through the documentation and here are two examples: one that documents success case and one that shows what is happening when there is an error.

In a sense the GIL (or actually GVL as it's called in current ruby versions) has already been removed for ruby.

It's only the original MRI Ruby that still has it several over Ruby implementations already removed it. e.g. JRuby.

Concurrent-Ruby[1] is probably a good place to start if you want to work with GVL free ruby on JRuby. It's quite well supported and is currently used by Rails.

If you just want async or non-blocking IO I'd take a look at the Async Gem[2]. It looks pretty solid in Ruby > 3.0 and it's been invited by Matz to be part of the stdlib, which I think is a pretty good endorsement.

For MRI itself I don't think it's likely they'll ever remove the GVL. Ractors are probably a better solution for CPU concurrency in the long run, although I think they're pretty experimental currently.

1. https://github.com/ruby-concurrency/concurrent-ruby

I like how the article exposes you to tools to prove/disprove the problem. I would have hoped it introduced to tools like concurrent ruby and the use of atomics like u/Freeky already mentioned though.

Using conccurrent-ruby, how can I execute a set of promises and then get the results?

I'm trying to add a watermark to images as they are uploaded before they're saved. I'm using the following code(borrowed heavily from the example here)

Photein uses Minimagick for image management and conversion. Have you had a look at ruby-vips? From what I hear it should be better/faster/use less memory than ImageMagick, especially for the standard image formats.

A special mention to ruby-vips, which is (along with maybe php, I guess) one of the few language bindings maintained by the author of libvips, which should become the defacto image processing manipulation library.