activestorage-aliyun VS Sidekiq

aliyun, qiniu, tencent cloud storage

- Support [aliyun](https://github.com/huacnlee/activestorage-aliyun), [qiniu](https://rubygems.org/gems/activestorage_qiniu) cloud storage

Uses Sidekiq's "fake" testing mode which will allow you to run the jobs explicitly

The code above isn't idempotent. If you run it twice, it will create two copies, which is probably not what you intended. Why is this important? Because most backend job processors like Sidekiq don't make any guarantees that your jobs will run exactly once.

There are several workarounds for this, like the magnificent Acidic Job gem or Sidekiq Pro/Enterprise features around enhanced reliability and unique jobs. Still, if they occur, bugs related to missing jobs and/or job idempotency are hard to track down and even harder to fix.

I forgot to mention that reading code is also a good way to learn how to write code, it's like inspiration. Check repos of some gems you like. For example sidekiq https://github.com/sidekiq/sidekiq/tree/main/lib/sidekiq Or minitest https://github.com/minitest/minitest/tree/master/lib/minitest

Starting service with 'bundle exec sidekiq' Mar 23 04:44:14 PM 2023-03-23T20:44:14.698Z pid=68 tid=2ks INFO: Booted Rails 7.0.4.2 application in production environment Mar 23 04:44:14 PM 2023-03-23T20:44:14.699Z pid=68 tid=2ks INFO: Running in ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] Mar 23 04:44:14 PM 2023-03-23T20:44:14.699Z pid=68 tid=2ks INFO: See LICENSE and the LGPL-3.0 for licensing details. Mar 23 04:44:14 PM 2023-03-23T20:44:14.699Z pid=68 tid=2ks INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org Mar 23 04:44:14 PM 2023-03-23T20:44:14.699Z pid=68 tid=2ks INFO: Sidekiq 7.0.7 connecting to Redis with options {:size=>5, :pool_name=>"internal", :url=>"redis://red-cgc7iqo2qv267ubpeaf0:6379"} Mar 23 04:44:14 PM getaddrinfo: Name or service not known Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/3.1.0/socket.rb:227:in `getaddrinfo' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/3.1.0/socket.rb:227:in `foreach' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/3.1.0/socket.rb:632:in `tcp' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client/ruby_connection.rb:111:in `connect' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client/ruby_connection.rb:49:in `initialize' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:671:in `new' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:671:in `block in connect' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client/middlewares.rb:12:in `connect' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:670:in `connect' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:657:in `raw_connection' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:624:in `ensure_connected' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client.rb:209:in `call' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/redis-client-0.14.0/lib/redis_client/decorator.rb:26:in `call' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/lib/sidekiq/config.rb:137:in `block in redis_info' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/lib/sidekiq/config.rb:156:in `block in redis' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/connection_pool-2.3.0/lib/connection_pool.rb:65:in `block (2 levels) in with' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `handle_interrupt' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `block in with' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `handle_interrupt' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `with' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/lib/sidekiq/config.rb:153:in `redis' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/lib/sidekiq/config.rb:136:in `redis_info' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/lib/sidekiq/cli.rb:75:in `run' Mar 23 04:44:14 PM /opt/render/project/.gems/ruby/3.1.0/gems/sidekiq-7.0.7/bin/sidekiq:31:in `' Mar 23 04:44:14 PM /opt/render/project/.gems/bin/sidekiq:27:in `load' Mar 23 04:44:14 PM /opt/render/project/.gems/bin/sidekiq:27:in `' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:492:in `exec' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:34:in `dispatch' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:28:in `start' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/bundler-2.4.9/exe/bundle:45:in `block in ' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:117:in `with_friendly_errors' Mar 23 04:44:14 PM /opt/render/project/rubies/ruby-3.1.3/lib/ruby/gems/3.1.0/gems/bundler-2.4.9/exe/bundle:33:in `' Mar 23 04:44:14 PM /opt/render/project/.gems/bin/bundle:113:in `load' Mar 23 04:44:14 PM /opt/render/project/.gems/bin/bundle:113:in `'

Background jobs are a handy tool for this, especially since Sidekiq uses retry and exponential backoff on error which handles recoverable like timeouts splendidly. The rate limit of this api (or the risk of DOSing it depending on why it's sluggish) will be the major consideration for this approach and may require some sort of throttle in your jobs to keep too many of them from running at once. Based on what you said about "if there's a running query" I'm guessing you're referring to that style of problem; other ways you might handle it could be to use a dedicated work queue with only a single worker, or to use some sort of connection pool to gate access to the slow service (esp. if other workers or the rails code ever has to access it). Those obstacles aside, queueing up tens or hundreds of jobs if that's what's necessary isn't really that big a deal if that's the appropriate scale & batch size for your problem.

What is more, there is already a quite popular, and beloved in the Ruby community, project named Sidekiq - https://github.com/sidekiq/sidekiq.

Not sure about EKS on AWS but on GCP spot instances have a 30s graceful shutdown which is enough for most of our jobs to complete, otherwise they'll be picked up again by the next worker since we use Sidekiq Pro and its https://github.com/sidekiq/sidekiq/wiki/Reliability feature

Sidekiq is a background job processor for Ruby applications, which is useful for running time-consuming tasks without blocking the main thread. To install Sidekiq, add it to your Gemfile: