Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Background jobs are a handy tool for this, especially since Sidekiq uses retry and exponential backoff on error which handles recoverable like timeouts splendidly. The rate limit of this api (or the risk of DOSing it depending on why it's sluggish) will be the major consideration for this approach and may require some sort of throttle in your jobs to keep too many of them from running at once. Based on what you said about "if there's a running query" I'm guessing you're referring to that style of problem; other ways you might handle it could be to use a dedicated work queue with only a single worker, or to use some sort of connection pool to gate access to the slow service (esp. if other workers or the rails code ever has to access it). Those obstacles aside, queueing up tens or hundreds of jobs if that's what's necessary isn't really that big a deal if that's the appropriate scale & batch size for your problem.
I am also noticing that there's a gem providing similar functionality: https://github.com/breamware/sidekiq-batch