Concurrent Ruby VS Sequel

Let’s say we need to traverse through thousands of files in our S3 Storage in a Ruby app. Let’s say we have a bunch of logs there that we need to read every day and process. If we just use a straightforward approach, like opening, reading, and processing every file one by one, our solution will work, but It will take a lot of time to process. So we need to improve the speed. Here ruby-concurrent gem is our helper https://github.com/ruby-concurrency/concurrent-ruby. Why do we need this gem? Because It’s simpler to use than Threads and this gem has a lot more features in It. In this article, we will use Concurrent::Promises.future as the most common use of concurrent code. Because reading a file from S3 is an IO operation, we can get a huge benefit in speed if we gonna use concurrent code doing HTTP requests. Remember that concurrency will not give you speed improvements if in every Promise or Thread you will do any calculations. Because of Ruby GIL, every thread will be blocked until calculations are finished.

concurrent-ruby - to add async requests ability. https://github.com/ruby-concurrency/concurrent-ruby

After this, I took a look at the semaphore class in the popular library, concurrent-ruby to see how they implement it, and I learnt about something new: condition variables. And Ruby comes with this included!

https://github.com/ruby-concurrency/concurrent-ruby has great docs if someone is looking for alternatives.

I wonder how this would compare to using concurrent-ruby under ruby 2.7, especially in a real-world setting (where the calls are actually to external services that return and buffer data, instead of just sleep). The author says that he's felt that ruby threads "feel easy to mess up," but I've found that concurrent-ruby makes it pretty simple, and performant enough even with the GIL.

As I could not find a clear example about how to rescue exceptions from Concurrent::Promises (part of the Concurrent Ruby gem ) I read through the documentation and here are two examples: one that documents success case and one that shows what is happening when there is an error.

In a sense the GIL (or actually GVL as it's called in current ruby versions) has already been removed for ruby.

It's only the original MRI Ruby that still has it several over Ruby implementations already removed it. e.g. JRuby.

Concurrent-Ruby[1] is probably a good place to start if you want to work with GVL free ruby on JRuby. It's quite well supported and is currently used by Rails.

If you just want async or non-blocking IO I'd take a look at the Async Gem[2]. It looks pretty solid in Ruby > 3.0 and it's been invited by Matz to be part of the stdlib, which I think is a pretty good endorsement.

For MRI itself I don't think it's likely they'll ever remove the GVL. Ractors are probably a better solution for CPU concurrency in the long run, although I think they're pretty experimental currently.

1. https://github.com/ruby-concurrency/concurrent-ruby

I like how the article exposes you to tools to prove/disprove the problem. I would have hoped it introduced to tools like concurrent ruby and the use of atomics like u/Freeky already mentioned though.

Using conccurrent-ruby, how can I execute a set of promises and then get the results?

While Ruby is not this famous anymore, I still wanted the stack in my architecture. I eschewed Ruby on Rails in favor of the leaner Sinatra framework. I use sequel for database access. The dynamic nature of the language was a bit of a hurdle, which is why it took me more time to develop my service than with Go.

Some of the most enlightening books I’ve read when I was first learning Ruby were Text Processing in Ruby, and Building Awesome Command Line Apps in Ruby 2. They each reveal certain features and perspectives that work towards this end, such as text parsing moves, Ruby flags to help you build shell 1-liners you can pipe against, and features with stdio beyond just printing to stdout.

Then add in something like Pry or Irb, where you are able to build castles in your sandbox.

Most of my data exploration happens in Pry.

A final book I’ll toss out is Data Science at the Command Line, in particular the first 40 or so pages. They highlight the amount of tooling that exists that’s just python shell scripts posing as bins. (Ruby of course has every bit of the same potential.) I had always been aware of this, but I found the way it was presented to be very inspirational, and largely transformed how I work with data.

A good practical example I use regularly is: I have a project set up that keeps connection strings for ten or so SQL Server DBs that I regularly interact with. I have constants defined to expedite connections. The [Sequel library](https://sequel.jeremyevans.net/) is absolutely delightful to use. I have a `bin/console` file that sets up a pry session hooking up the default environment and tools I like to work with. Now it’s very easy to find tables with certain names, schemas, containing certain data, certain sprocs, mass update definitions across our entire system.

```

Thea answer to your prayers already exists: http://sequel.jeremyevans.net/.

By far the best database toolkit (ORM, query builder, migration engine) I have seen for any programming language.

Ruby sequel (http://sequel.jeremyevans.net/) is the only library where you can combine classic ORM Model bases usage, with a more raw query builder "just get me all the data into plain objects". You'll never need anything again in your career life.

If you want a db tool which can be an ORM for your app, and drop down to a lower level dsl, while targeting specific features of the databases it supports, + having a "composable superset for building queries", there's [ruby sequel](http://sequel.jeremyevans.net/), which is the best tool of the kind you'll get for any proglang. Everything the author wants, minus the typrchecking perhaps, which is IMO shooting at the stars.