sidekiq-limit-fetch-3.4.0/0000755000175000017500000000000013043067544014404 5ustar pravipravisidekiq-limit-fetch-3.4.0/README.md0000644000175000017500000000776013043067544015675 0ustar pravipravi## Description Sidekiq strategy to support a granular queue control – limiting, pausing, blocking, querying. [![Build Status](https://secure.travis-ci.org/brainopia/sidekiq-limit_fetch.png)](http://travis-ci.org/brainopia/sidekiq-limit_fetch) [![Gem Version](https://badge.fury.io/rb/sidekiq-limit_fetch.png)](http://badge.fury.io/rb/sidekiq-limit_fetch) [![Dependency Status](https://gemnasium.com/brainopia/sidekiq-limit_fetch.png)](https://gemnasium.com/brainopia/sidekiq-limit_fetch) [![Code Climate](https://codeclimate.com/github/brainopia/sidekiq-limit_fetch.png)](https://codeclimate.com/github/brainopia/sidekiq-limit_fetch) ## Installation Add this line to your application's Gemfile: gem 'sidekiq-limit_fetch' ### Requirements **Important note:** At this moment, `sidekiq-limit_fetch` is incompatible with - sidekiq pro's `reliable_fetch` - `sidekiq-rate-limiter` - any other plugin that rewrites fetch strategy of sidekiq. ## Usage ### Limits Specify limits which you want to place on queues inside sidekiq.yml: ```yaml :limits: queue_name1: 5 queue_name2: 10 ``` Or set it dynamically in your code: ```ruby Sidekiq::Queue['queue_name1'].limit = 5 Sidekiq::Queue['queue_name2'].limit = 10 ``` In these examples, tasks for the ```queue_name1``` will be run by at most 5 workers at the same time and the ```queue_name2``` will have no more than 10 workers simultaneously. Ability to set limits dynamically allows you to resize worker distribution among queues any time you want. ### Limits per process If you use multiple sidekiq processes then you can specify limits per process: ```yaml :process_limits: queue_name: 2 ``` Or set it in your code: ```ruby Sidekiq::Queue['queue_name'].process_limit = 2 ``` ### Busy workers by queue You can see how many workers currently handling a queue: ```ruby Sidekiq::Queue['name'].busy # number of busy workers ``` ### Pauses You can also pause your queues temporarely. Upon continuing their limits will be preserved. ```ruby Sidekiq::Queue['name'].pause # prevents workers from running tasks from this queue Sidekiq::Queue['name'].paused? # => true Sidekiq::Queue['name'].unpause # allows workers to use the queue Sidekiq::Queue['name'].pause_for_ms(1000) # will pause for a second ``` ### Blocking queue mode If you use strict queue ordering (it will be used if you don't specify queue weights) then you can set blocking status for queues. It means if a blocking queue task is executing then no new task from lesser priority queues will be ran. Eg, ```yaml :queues: - a - b - c :blocking: - b ``` In this case when a task for `b` queue is ran no new task from `c` queue will be started. You can also enable and disable blocking mode for queues on the fly: ```ruby Sidekiq::Queue['name'].block Sidekiq::Queue['name'].blocking? # => true Sidekiq::Queue['name'].unblock ``` ### Advanced blocking queues You can also block on array of queues. It means when any of them is running only queues higher and queues from their blocking group can run. It will be easier to understand with an example: ```yaml :queues: - a - b - c - d :blocking: - [b, c] ``` In this case tasks from `d` will be blocked when a task from queue `b` or `c` is executed. You can dynamically set exceptions for queue blocking: ```ruby Sidekiq::Queue['queue1'].block_except 'queue2' ``` ### Dynamic queues You can support dynamic queues (that are not listed in sidekiq.yml but that have tasks pushed to them (usually with `Sidekiq::Client.push`)). To use this mode you need to specify a following line in sidekiq.yml: ```yaml :dynamic: true ``` Dynamic queues will be ran at the lowest priority. ### Maintenance If you use ```flushdb```, restart the sidekiq process to re-populate the dynamic configuration. ### Thanks Sponsored by Evil Martians sidekiq-limit-fetch-3.4.0/bench/0000755000175000017500000000000013043067544015463 5ustar pravipravisidekiq-limit-fetch-3.4.0/bench/compare.rb0000644000175000017500000000205713043067544017442 0ustar pravipravirequire 'benchmark' require 'sidekiq/cli' require 'sidekiq/api' total = (ARGV.shift || 50).to_i concurrency = ARGV.shift || 1 limit = ARGV.shift if limit limit = nil if limit == 'nil' $:.unshift File.expand_path '../lib' require 'sidekiq-limit_fetch' Sidekiq::Queue['inline'].limit = limit Sidekiq.redis {|it| it.del 'limit_fetch:probed:inline' } Sidekiq::LimitFetch::Queues.send(:define_method, :set) {|*| } end Sidekiq::Queue.new('inline').clear class FastJob include Sidekiq::Worker sidekiq_options queue: :inline def perform(i) puts "job N#{i} is finished" end end class FinishJob include Sidekiq::Worker sidekiq_options queue: :inline def perform Process.kill 'INT', 0 end end total.times {|i| FastJob.perform_async i+1 } FinishJob.perform_async Sidekiq::CLI.instance.tap do |cli| %w(validate! boot_system).each {|stub| cli.define_singleton_method(stub) {}} cli.parse ['-q inline', '-q other', "-c #{concurrency}"] puts Benchmark.measure { begin cli.run rescue Exception end } end sidekiq-limit-fetch-3.4.0/demo/0000755000175000017500000000000013043067544015330 5ustar pravipravisidekiq-limit-fetch-3.4.0/demo/config/0000755000175000017500000000000013043067544016575 5ustar pravipravisidekiq-limit-fetch-3.4.0/demo/config/environments/0000755000175000017500000000000013043067544021324 5ustar pravipravisidekiq-limit-fetch-3.4.0/demo/config/environments/development.rb0000644000175000017500000000050013043067544024166 0ustar pravipraviDemo::Application.configure do config.cache_classes = false config.eager_load = false config.consider_all_requests_local = true config.action_controller.perform_caching = false config.action_mailer.raise_delivery_errors = false config.active_support.deprecation = :log config.assets.debug = true end sidekiq-limit-fetch-3.4.0/demo/config/boot.rb0000644000175000017500000000020513043067544020062 0ustar pravipraviENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__) require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE']) sidekiq-limit-fetch-3.4.0/demo/config/application.rb0000644000175000017500000000036313043067544021427 0ustar pravipravirequire File.expand_path('../boot', __FILE__) require 'action_controller/railtie' require 'action_mailer/railtie' require 'sprockets/railtie' Bundler.require(:default, Rails.env) module Demo Application = Class.new Rails::Application end sidekiq-limit-fetch-3.4.0/demo/config/environment.rb0000644000175000017500000000012313043067544021462 0ustar pravipravirequire File.expand_path('../application', __FILE__) Demo::Application.initialize! sidekiq-limit-fetch-3.4.0/demo/README.md0000644000175000017500000000161613043067544016613 0ustar pravipraviThis is a demo rails app with a configured sidekiq-limit_fetch. Its purpose is to check whether plugin works in certain situations. Application is preconfigured with two workers: - `app/workers/fast_worker.rb` which does `sleep 0.2` - `app/workers/slow_worker.rb` which does `sleep 1` There is also a rake task which can be invoked as `bundle exec rake demo:limit`: - it prefills sidekiq tasks ```ruby 100.times do SlowWorker.perform_async FastWorker.perform_async end ``` - sets sidekiq config ```yaml :verbose: false :concurrency: 4 :queues: - slow - fast :limits: slow: 1 ``` - and launches a sidekiq admin page with overview of queues in browser. The page is set to live-poll so effects of limits can be seen directly. To change simulation modify `Rakefile` or workers. Any bugs related to the plugin should be demonstrated with a reproduction from this base app. sidekiq-limit-fetch-3.4.0/demo/Rakefile0000644000175000017500000000370613043067544017003 0ustar pravipravirequire File.expand_path('../config/application', __FILE__) Demo::Application.load_tasks namespace :demo do task limit: :environment do puts '=> Creating sidekiq tasks' 100.times do SlowWorker.perform_async FastWorker.perform_async end run_sidekiq_monitoring run_sidekiq_workers config: <<-YAML :verbose: false :concurrency: 4 :queues: - slow - fast :limits: slow: 1 YAML end task blocking: :environment do puts '=> Creating sidekiq tasks' AWorker.perform_async BWorker.perform_async CWorker.perform_async run_sidekiq_monitoring run_sidekiq_workers config: <<-YAML :verbose: false :concurrency: 4 :queues: - a - b - c :blocking: - a YAML end task advanced_blocking: :environment do puts '=> Creating sidekiq tasks' AWorker.perform_async BWorker.perform_async CWorker.perform_async run_sidekiq_monitoring run_sidekiq_workers config: <<-YAML :verbose: false :concurrency: 4 :queues: - a - b - c :blocking: - [a, b] YAML end def with_sidekiq_config(config) whitespace_offset = config[/\A */].size config.gsub! /^ {#{whitespace_offset}}/, '' puts "=> Use sidekiq config:\n#{config}" File.write 'config/sidekiq.yml', config yield ensure FileUtils.rm 'config/sidekiq.yml' end def run_sidekiq_monitoring require 'sidekiq/web' Thread.new do Rack::Server.start app: Sidekiq::Web, Port: 3000 end sleep 1 Launchy.open 'http://127.0.0.1:3000/workers?poll=true' end def run_sidekiq_workers(options) require 'sidekiq/cli' cli = Sidekiq::CLI.instance %w(validate! boot_system).each do |stub| cli.define_singleton_method(stub) {} end with_sidekiq_config options[:config] do cli.send :setup_options, [] end cli.run end end sidekiq-limit-fetch-3.4.0/demo/app/0000755000175000017500000000000013043067544016110 5ustar pravipravisidekiq-limit-fetch-3.4.0/demo/app/workers/0000755000175000017500000000000013043067544017604 5ustar pravipravisidekiq-limit-fetch-3.4.0/demo/app/workers/a_worker.rb0000644000175000017500000000015213043067544021740 0ustar pravipraviclass AWorker include Sidekiq::Worker sidekiq_options queue: :a def perform sleep 10 end end sidekiq-limit-fetch-3.4.0/demo/app/workers/slow_worker.rb0000644000175000017500000000015713043067544022511 0ustar pravipraviclass SlowWorker include Sidekiq::Worker sidekiq_options queue: :slow def perform sleep 1 end end sidekiq-limit-fetch-3.4.0/demo/app/workers/c_worker.rb0000644000175000017500000000015313043067544021743 0ustar pravipraviclass CWorker include Sidekiq::Worker sidekiq_options queue: :c def perform sleep 10 end end sidekiq-limit-fetch-3.4.0/demo/app/workers/b_worker.rb0000644000175000017500000000015213043067544021741 0ustar pravipraviclass BWorker include Sidekiq::Worker sidekiq_options queue: :b def perform sleep 10 end end sidekiq-limit-fetch-3.4.0/demo/app/workers/fast_worker.rb0000644000175000017500000000016113043067544022455 0ustar pravipraviclass FastWorker include Sidekiq::Worker sidekiq_options queue: :fast def perform sleep 0.2 end end sidekiq-limit-fetch-3.4.0/demo/Gemfile0000644000175000017500000000024713043067544016626 0ustar pravipravisource 'https://rubygems.org' gem 'rails' gem 'sinatra' gem 'celluloid' gem 'launchy' gem 'sidekiq', github: 'mperham/sidekiq' gem 'sidekiq-limit_fetch', path: '..' sidekiq-limit-fetch-3.4.0/lib/0000755000175000017500000000000013043067544015152 5ustar pravipravisidekiq-limit-fetch-3.4.0/lib/sidekiq/0000755000175000017500000000000013043067544016603 5ustar pravipravisidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/0000755000175000017500000000000013043067544021072 5ustar pravipravisidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/queues.rb0000644000175000017500000000506213043067544022731 0ustar pravipravimodule Sidekiq::LimitFetch::Queues extend self THREAD_KEY = :acquired_queues def start(options) @queues = options[:queues] @dynamic = options[:dynamic] @limits = options[:limits] || {} @process_limits = options[:process_limits] || {} @blocks = options[:blocking] || [] options[:strict] ? strict_order! : weighted_order! apply_process_limit_to_queues apply_limit_to_queues apply_blocks_to_queues end def acquire selector.acquire(ordered_queues, namespace) .tap {|it| save it } .map {|it| "queue:#{it}" } end def release_except(full_name) queues = restore queues.delete full_name[/queue:(.*)/, 1] if full_name Sidekiq::LimitFetch.redis_retryable do selector.release queues, namespace end end def dynamic? @dynamic end def add(queues) queues.each do |queue| unless @queues.include? queue apply_process_limit_to_queue(queue) apply_limit_to_queue(queue) @queues.push queue end end end def strict_order! @queues.uniq! def ordered_queues; @queues end end def weighted_order! def ordered_queues; @queues.shuffle.uniq end end def namespace @namespace ||= Sidekiq.redis do |it| if it.respond_to?(:namespace) and it.namespace "#{it.namespace}:" else '' end end end private def apply_process_limit_to_queues @queues.uniq.each do |queue_name| apply_process_limit_to_queue(queue_name) end end def apply_process_limit_to_queue(queue_name) queue = Sidekiq::Queue[queue_name] queue.process_limit = @process_limits[queue_name.to_s] || @process_limits[queue_name.to_sym] end def apply_limit_to_queues @queues.uniq.each do |queue_name| apply_limit_to_queue(queue_name) end end def apply_limit_to_queue(queue_name) queue = Sidekiq::Queue[queue_name] unless queue.limit_changed? queue.limit = @limits[queue_name.to_s] || @limits[queue_name.to_sym] end end def apply_blocks_to_queues @queues.uniq.each do |queue_name| Sidekiq::Queue[queue_name].unblock end @blocks.to_a.each do |it| if it.is_a? Array it.each {|name| Sidekiq::Queue[name].block_except it } else Sidekiq::Queue[it].block end end end def selector Sidekiq::LimitFetch::Global::Selector end def save(queues) Thread.current[THREAD_KEY] = queues end def restore Thread.current[THREAD_KEY] || [] ensure Thread.current[THREAD_KEY] = nil end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/unit_of_work.rb0000644000175000017500000000075013043067544024126 0ustar pravipravimodule Sidekiq class LimitFetch::UnitOfWork < BasicFetch::UnitOfWork def initialize(queue, job) super redis_retryable { Queue[queue_name].increase_busy } end def acknowledge redis_retryable { Queue[queue_name].decrease_busy } redis_retryable { Queue[queue_name].release } end def requeue super acknowledge end private def redis_retryable(&block) Sidekiq::LimitFetch.redis_retryable(&block) end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/global/0000755000175000017500000000000013043067544022332 5ustar pravipravisidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/global/selector.rb0000644000175000017500000000735213043067544024506 0ustar pravipravimodule Sidekiq::LimitFetch::Global module Selector extend self MUTEX_FOR_UUID = Mutex.new def acquire(queues, namespace) redis_eval :acquire, [namespace, uuid, queues] end def release(queues, namespace) redis_eval :release, [namespace, uuid, queues] end def uuid # - if we'll remove "@uuid ||=" from inside of mutex # then @uuid can be overwritten # - if we'll remove "@uuid ||=" from outside of mutex # then each read will lead to mutex @uuid ||= MUTEX_FOR_UUID.synchronize { @uuid || SecureRandom.uuid } end private def redis_eval(script_name, args) Sidekiq.redis do |it| begin it.evalsha send("redis_#{script_name}_sha"), argv: args rescue Redis::CommandError => error raise unless error.message.include? 'NOSCRIPT' it.eval send("redis_#{script_name}_script"), argv: args end end end def redis_acquire_sha @acquire_sha ||= Digest::SHA1.hexdigest redis_acquire_script end def redis_release_sha @release_sha ||= Digest::SHA1.hexdigest redis_release_script end def redis_acquire_script <<-LUA local namespace = table.remove(ARGV, 1)..'limit_fetch:' local worker_name = table.remove(ARGV, 1) local queues = ARGV local available = {} local unblocked = {} local locks local process_locks local blocking_mode for _, queue in ipairs(queues) do if not blocking_mode or unblocked[queue] then local probed_key = namespace..'probed:'..queue local pause_key = namespace..'pause:'..queue local limit_key = namespace..'limit:'..queue local process_limit_key = namespace..'process_limit:'..queue local block_key = namespace..'block:'..queue local paused, limit, process_limit, can_block = unpack(redis.call('mget', pause_key, limit_key, process_limit_key, block_key )) if not paused then limit = tonumber(limit) process_limit = tonumber(process_limit) if can_block or limit then locks = redis.call('llen', probed_key) end if process_limit then local all_locks = redis.call('lrange', probed_key, 0, -1) process_locks = 0 for _, process in ipairs(all_locks) do if process == worker_name then process_locks = process_locks + 1 end end end if not blocking_mode then blocking_mode = can_block and locks > 0 end if blocking_mode and can_block ~= 'true' then for unblocked_queue in string.gmatch(can_block, "[^,]+") do unblocked[unblocked_queue] = true end end if (not limit or limit > locks) and (not process_limit or process_limit > process_locks) then redis.call('rpush', probed_key, worker_name) table.insert(available, queue) end end end end return available LUA end def redis_release_script <<-LUA local namespace = table.remove(ARGV, 1)..'limit_fetch:' local worker_name = table.remove(ARGV, 1) local queues = ARGV for _, queue in ipairs(queues) do local probed_key = namespace..'probed:'..queue redis.call('lrem', probed_key, 1, worker_name) end LUA end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/global/semaphore.rb0000644000175000017500000000716113043067544024647 0ustar pravipravimodule Sidekiq::LimitFetch::Global class Semaphore PREFIX = 'limit_fetch' attr_reader :local_busy def initialize(name) @name = name @lock = Mutex.new @local_busy = 0 end def limit value = redis {|it| it.get "#{PREFIX}:limit:#@name" } value.to_i if value end def limit=(value) @limit_changed = true if value redis {|it| it.set "#{PREFIX}:limit:#@name", value } else redis {|it| it.del "#{PREFIX}:limit:#@name" } end end def limit_changed? @limit_changed end def process_limit value = redis {|it| it.get "#{PREFIX}:process_limit:#@name" } value.to_i if value end def process_limit=(value) if value redis {|it| it.set "#{PREFIX}:process_limit:#@name", value } else redis {|it| it.del "#{PREFIX}:process_limit:#@name" } end end def acquire Selector.acquire([@name], namespace).size > 0 end def release redis {|it| it.lrem "#{PREFIX}:probed:#@name", 1, Selector.uuid } end def busy redis {|it| it.llen "#{PREFIX}:busy:#@name" } end def busy_processes redis {|it| it.lrange "#{PREFIX}:busy:#@name", 0, -1 } end def increase_busy increase_local_busy redis {|it| it.rpush "#{PREFIX}:busy:#@name", Selector.uuid } end def decrease_busy decrease_local_busy redis {|it| it.lrem "#{PREFIX}:busy:#@name", 1, Selector.uuid } end def probed redis {|it| it.llen "#{PREFIX}:probed:#@name" } end def probed_processes redis {|it| it.lrange "#{PREFIX}:probed:#@name", 0, -1 } end def pause redis {|it| it.set "#{PREFIX}:pause:#@name", true } end def pause_for_ms ms redis {|it| it.psetex "#{PREFIX}:pause:#@name", ms, true } end def unpause redis {|it| it.del "#{PREFIX}:pause:#@name" } end def paused? redis {|it| it.get "#{PREFIX}:pause:#@name" } end def block redis {|it| it.set "#{PREFIX}:block:#@name", true } end def block_except(*queues) raise ArgumentError if queues.empty? redis {|it| it.set "#{PREFIX}:block:#@name", queues.join(',') } end def unblock redis {|it| it.del "#{PREFIX}:block:#@name" } end def blocking? redis {|it| it.get "#{PREFIX}:block:#@name" } end def increase_local_busy @lock.synchronize { @local_busy += 1 } end def decrease_local_busy @lock.synchronize { @local_busy -= 1 } end def local_busy? @local_busy > 0 end def explain <<-END.gsub(/^ {8}/, '') Current sidekiq process: #{Selector.uuid} All processes: #{Monitor.all_processes.join "\n"} Stale processes: #{Monitor.old_processes.join "\n"} Locked queue processes: #{probed_processes.sort.join "\n"} Busy queue processes: #{busy_processes.sort.join "\n"} Limit: #{limit.inspect} Process limit: #{process_limit.inspect} Blocking: #{blocking?} END end def remove_locks_except!(processes) locked_processes = probed_processes.uniq (locked_processes - processes).each do |dead_process| remove_lock! dead_process end end def remove_lock!(process) redis do |it| it.lrem "#{PREFIX}:probed:#@name", 0, process it.lrem "#{PREFIX}:busy:#@name", 0, process end end private def redis(&block) Sidekiq.redis(&block) end def namespace Sidekiq::LimitFetch::Queues.namespace end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/global/monitor.rb0000644000175000017500000000312113043067544024343 0ustar pravipravimodule Sidekiq::LimitFetch::Global module Monitor extend self HEARTBEAT_PREFIX = 'limit:heartbeat:' PROCESS_SET = 'limit:processes' HEARTBEAT_TTL = 20 REFRESH_TIMEOUT = 5 def start!(ttl=HEARTBEAT_TTL, timeout=REFRESH_TIMEOUT) Thread.new do loop do Sidekiq::LimitFetch.redis_retryable do add_dynamic_queues update_heartbeat ttl invalidate_old_processes end sleep timeout end end end def all_processes Sidekiq.redis {|it| it.smembers PROCESS_SET } end def old_processes all_processes.reject do |process| Sidekiq.redis {|it| it.get heartbeat_key process } end end def remove_old_processes! Sidekiq.redis do |it| old_processes.each {|process| it.srem PROCESS_SET, process } end end def add_dynamic_queues queues = Sidekiq::LimitFetch::Queues queues.add Sidekiq::Queue.all.map(&:name) if queues.dynamic? end private def update_heartbeat(ttl) Sidekiq.redis do |it| it.multi do it.set heartbeat_key, true it.sadd PROCESS_SET, Selector.uuid it.expire heartbeat_key, ttl end end end def invalidate_old_processes Sidekiq.redis do |it| remove_old_processes! processes = all_processes Sidekiq::Queue.instances.each do |queue| queue.remove_locks_except! processes end end end def heartbeat_key(process=Selector.uuid) HEARTBEAT_PREFIX + process end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch/instances.rb0000644000175000017500000000044113043067544023405 0ustar pravipravimodule Sidekiq::LimitFetch::Instances def self.extended(klass) klass.instance_variable_set :@instances, {} end def new(*args) @instances[args] ||= super end alias [] new def instances @instances.values end def reset_instances! @instances = {} end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/limit_fetch.rb0000644000175000017500000000220613043067544021417 0ustar pravipravirequire 'forwardable' require 'sidekiq' require 'sidekiq/manager' require 'sidekiq/api' module Sidekiq::LimitFetch autoload :UnitOfWork, 'sidekiq/limit_fetch/unit_of_work' require_relative 'limit_fetch/instances' require_relative 'limit_fetch/queues' require_relative 'limit_fetch/global/semaphore' require_relative 'limit_fetch/global/selector' require_relative 'limit_fetch/global/monitor' require_relative 'extensions/queue' require_relative 'extensions/manager' extend self def new(_) self end def retrieve_work queue, job = redis_brpop(Queues.acquire) Queues.release_except(queue) UnitOfWork.new(queue, job) if job end def bulk_requeue(*args) Sidekiq::BasicFetch.bulk_requeue(*args) end def redis_retryable yield rescue Redis::BaseConnectionError sleep 1 retry end private TIMEOUT = Sidekiq::BasicFetch::TIMEOUT def redis_brpop(queues) if queues.empty? sleep TIMEOUT # there are no queues to handle, so lets sleep [] # and return nothing else redis_retryable { Sidekiq.redis { |it| it.brpop *queues, TIMEOUT } } end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/extensions/0000755000175000017500000000000013043067544021002 5ustar pravipravisidekiq-limit-fetch-3.4.0/lib/sidekiq/extensions/queue.rb0000644000175000017500000000115213043067544022452 0ustar pravipravimodule Sidekiq class Queue extend LimitFetch::Instances, Forwardable attr_reader :rname def_delegators :lock, :limit, :limit=, :limit_changed?, :process_limit, :process_limit=, :acquire, :release, :pause, :pause_for_ms, :unpause, :block, :unblock, :paused?, :blocking?, :unblocked, :block_except, :probed, :busy, :increase_busy, :decrease_busy, :local_busy?, :explain, :remove_locks_except! def lock @lock ||= LimitFetch::Global::Semaphore.new name end end end sidekiq-limit-fetch-3.4.0/lib/sidekiq/extensions/manager.rb0000644000175000017500000000046713043067544022750 0ustar pravipraviclass Sidekiq::Manager module InitLimitFetch def initialize(options={}) options[:fetch] = Sidekiq::LimitFetch super end def start Sidekiq::LimitFetch::Queues.start options Sidekiq::LimitFetch::Global::Monitor.start! super end end prepend InitLimitFetch end sidekiq-limit-fetch-3.4.0/lib/sidekiq-limit_fetch.rb0000644000175000017500000000004713043067544021416 0ustar pravipravirequire_relative 'sidekiq/limit_fetch' sidekiq-limit-fetch-3.4.0/.travis.yml0000644000175000017500000000021513043067544016513 0ustar pravipravilanguage: ruby before_install: - gem install bundler - gem update bundler rvm: - 2.3.1 - 2.2.5 - jruby-head services: - redis-server sidekiq-limit-fetch-3.4.0/LICENSE.txt0000644000175000017500000000205113043067544016225 0ustar pravipraviCopyright (c) 2013 brainopia MIT License Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.sidekiq-limit-fetch-3.4.0/.gitignore0000644000175000017500000000003313043067544016370 0ustar pravipraviGemfile.lock pkg/ .bundle/ sidekiq-limit-fetch-3.4.0/sidekiq-limit_fetch.gemspec0000644000175000017500000000151213043067544021666 0ustar pravipraviGem::Specification.new do |gem| gem.name = 'sidekiq-limit_fetch' gem.version = '3.4.0' gem.license = 'MIT' gem.authors = 'brainopia' gem.email = 'brainopia@evilmartians.com' gem.summary = 'Sidekiq strategy to support queue limits' gem.homepage = 'https://github.com/brainopia/sidekiq-limit_fetch' gem.description = <<-DESCRIPTION Sidekiq strategy to restrict number of workers which are able to run specified queues simultaneously. DESCRIPTION gem.files = `git ls-files`.split($/) gem.test_files = gem.files.grep %r{^spec/} gem.require_paths = %w(lib) gem.add_dependency 'sidekiq', '>= 4' gem.add_development_dependency 'redis-namespace', '~> 1.5', '>= 1.5.2' gem.add_development_dependency 'rspec' gem.add_development_dependency 'rake' end sidekiq-limit-fetch-3.4.0/Rakefile0000644000175000017500000000033013043067544016045 0ustar pravipravirequire 'bundler/gem_tasks' require 'rspec/core/rake_task' RSpec::Core::RakeTask.new task :default do rspec = Rake::Task[:spec] rspec.invoke ENV['namespace'] = 'namespace' rspec.reenable rspec.invoke end sidekiq-limit-fetch-3.4.0/Gemfile0000644000175000017500000000006513043067544015700 0ustar pravipravisource 'https://rubygems.org' gemspec gem 'sidekiq' sidekiq-limit-fetch-3.4.0/.rspec0000644000175000017500000000003613043067544015520 0ustar pravipravi--require spec_helper --color sidekiq-limit-fetch-3.4.0/spec/0000755000175000017500000000000013043067544015336 5ustar pravipravisidekiq-limit-fetch-3.4.0/spec/sidekiq/0000755000175000017500000000000013043067544016767 5ustar pravipravisidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch/0000755000175000017500000000000013043067544021256 5ustar pravipravisidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch/semaphore_spec.rb0000644000175000017500000000301013043067544024572 0ustar pravipraviRSpec.describe 'semaphore' do let(:name) { 'default' } subject { Sidekiq::LimitFetch::Global::Semaphore.new name } it 'should have no limit by default' do expect(subject.limit).not_to be end it 'should set limit' do subject.limit = 4 expect(subject.limit).to eq 4 end it 'should acquire and count active tasks' do 3.times { subject.acquire } expect(subject.probed).to eq 3 end it 'should acquire tasks with regard to limit' do subject.limit = 4 6.times { subject.acquire } expect(subject.probed).to eq 4 end it 'should acquire tasks with regard to process limit' do subject.process_limit = 4 6.times { subject.acquire } expect(subject.probed).to eq 4 end it 'should release active tasks' do 6.times { subject.acquire } 3.times { subject.release } expect(subject.probed).to eq 3 end it 'should pause tasks' do 3.times { subject.acquire } subject.pause 2.times { subject.acquire } expect(subject.probed).to eq 3 2.times { subject.release } expect(subject.probed).to eq 1 end it 'should unpause tasks' do subject.pause 3.times { subject.acquire } subject.unpause 2.times { subject.acquire } expect(subject.probed).to eq 2 end it 'should pause tasks for a limited time' do 3.times { subject.acquire } subject.pause_for_ms 50 2.times { subject.acquire } expect(subject.probed).to eq 3 sleep(100.0 / 1000) 2.times { subject.acquire } expect(subject.probed).to eq 5 end end sidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch/global/0000755000175000017500000000000013043067544022516 5ustar pravipravisidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch/global/monitor_spec.rb0000644000175000017500000000152713043067544025551 0ustar pravipraviRSpec.describe Sidekiq::LimitFetch::Global::Monitor do let(:monitor) { described_class.start! ttl, timeout } let(:ttl) { 1 } let(:queue) { Sidekiq::Queue[name] } let(:name) { 'default' } before { monitor } after { monitor.kill } context 'old locks' do let(:timeout) { 0.5 } it 'should remove invalidated old locks' do 2.times { queue.acquire } sleep 2*ttl expect(queue.probed).to eq 2 allow(described_class).to receive(:update_heartbeat) sleep 2*ttl expect(queue.probed).to eq 0 end it 'should remove invalid locks' do 2.times { queue.acquire } allow(described_class).to receive(:update_heartbeat) Sidekiq.redis do |it| it.del Sidekiq::LimitFetch::Global::Monitor::PROCESS_SET end sleep 2*ttl expect(queue.probed).to eq 0 end end end sidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch/queues_spec.rb0000644000175000017500000000537413043067544024135 0ustar pravipraviRSpec.describe Sidekiq::LimitFetch::Queues do let(:queues) { %w[queue1 queue2] } let(:limits) {{ 'queue1' => 3 }} let(:strict) { true } let(:blocking) {} let(:process_limits) {{ 'queue2' => 3 }} let(:options) do { queues: queues, limits: limits, strict: strict, blocking: blocking, process_limits: process_limits } end before { subject.start options } it 'should acquire queues' do subject.acquire expect(Sidekiq::Queue['queue1'].probed).to eq 1 expect(Sidekiq::Queue['queue2'].probed).to eq 1 end it 'should acquire dynamically blocking queues' do subject.acquire expect(Sidekiq::Queue['queue1'].probed).to eq 1 expect(Sidekiq::Queue['queue2'].probed).to eq 1 Sidekiq::Queue['queue1'].block subject.acquire expect(Sidekiq::Queue['queue1'].probed).to eq 2 expect(Sidekiq::Queue['queue2'].probed).to eq 1 end it 'should block except given queues' do Sidekiq::Queue['queue1'].block_except 'queue2' subject.acquire expect(Sidekiq::Queue['queue1'].probed).to eq 1 expect(Sidekiq::Queue['queue2'].probed).to eq 1 Sidekiq::Queue['queue1'].block_except 'queue404' subject.acquire expect(Sidekiq::Queue['queue1'].probed).to eq 2 expect(Sidekiq::Queue['queue2'].probed).to eq 1 end it 'should release queues' do subject.acquire subject.release_except nil expect(Sidekiq::Queue['queue1'].probed).to eq 0 expect(Sidekiq::Queue['queue2'].probed).to eq 0 end it 'should release queues except selected' do subject.acquire subject.release_except 'queue:queue1' expect(Sidekiq::Queue['queue1'].probed).to eq 1 expect(Sidekiq::Queue['queue2'].probed).to eq 0 end it 'should release when no queues was acquired' do queues.each {|name| Sidekiq::Queue[name].pause } subject.acquire expect { subject.release_except nil }.not_to raise_exception end context 'blocking' do let(:blocking) { %w(queue1) } it 'should acquire blocking queues' do 3.times { subject.acquire } expect(Sidekiq::Queue['queue1'].probed).to eq 3 expect(Sidekiq::Queue['queue2'].probed).to eq 1 end end it 'should set limits' do subject expect(Sidekiq::Queue['queue1'].limit).to eq 3 expect(Sidekiq::Queue['queue2'].limit).not_to be end it 'should set process_limits' do subject expect(Sidekiq::Queue['queue2'].process_limit).to eq 3 end context 'without strict flag' do let(:strict) { false } it 'should retrieve weighted queues' do expect(subject.ordered_queues).to match_array(%w(queue1 queue2)) end end it 'with strict flag should retrieve strictly ordered queues' do expect(subject.ordered_queues).to eq %w(queue1 queue2) end end sidekiq-limit-fetch-3.4.0/spec/sidekiq/limit_fetch_spec.rb0000644000175000017500000000242513043067544022620 0ustar pravipraviThread.abort_on_exception = true RSpec.describe Sidekiq::LimitFetch do let(:options) {{ queues: queues, limits: limits }} let(:queues) { %w(queue1 queue1 queue2 queue2) } let(:limits) {{ 'queue1' => 1, 'queue2' => 2 }} before do subject::Queues.start options Sidekiq.redis do |it| it.del 'queue:queue1' it.lpush 'queue:queue1', 'task1' it.lpush 'queue:queue1', 'task2' it.expire 'queue:queue1', 30 end end it 'should acquire lock on queue for execution' do work = subject.retrieve_work expect(work.queue_name).to eq 'queue1' expect(work.job).to eq 'task1' expect(Sidekiq::Queue['queue1'].busy).to eq 1 expect(Sidekiq::Queue['queue2'].busy).to eq 0 expect(subject.retrieve_work).not_to be work.requeue expect(Sidekiq::Queue['queue1'].busy).to eq 0 expect(Sidekiq::Queue['queue2'].busy).to eq 0 work = subject.retrieve_work expect(work.job).to eq 'task1' expect(Sidekiq::Queue['queue1'].busy).to eq 1 expect(Sidekiq::Queue['queue2'].busy).to eq 0 expect(subject.retrieve_work).not_to be work.acknowledge expect(Sidekiq::Queue['queue1'].busy).to eq 0 expect(Sidekiq::Queue['queue2'].busy).to eq 0 work = subject.retrieve_work expect(work.job).to eq 'task2' end end sidekiq-limit-fetch-3.4.0/spec/sidekiq/extensions/0000755000175000017500000000000013043067544021166 5ustar pravipravisidekiq-limit-fetch-3.4.0/spec/sidekiq/extensions/queue_spec.rb0000644000175000017500000000440513043067544023654 0ustar pravipraviRSpec.describe Sidekiq::Queue do context 'singleton' do shared_examples :constructor do it 'with default name' do new_object = -> { described_class.send constructor } expect(new_object.call).to eq new_object.call end it 'with given name' do new_object = ->(name) { described_class.send constructor, name } expect(new_object.call('name')).to eq new_object.call('name') end end context '.new' do let(:constructor) { :new } it_behaves_like :constructor end context '.[]' do let(:constructor) { :[] } it_behaves_like :constructor end context '#lock' do let(:name) { 'example' } let(:queue) { Sidekiq::Queue[name] } it 'should be available' do expect(queue.acquire).to be end it 'should be pausable' do queue.pause expect(queue.acquire).not_to be end it 'should be continuable' do queue.pause queue.unpause expect(queue.acquire).to be end it 'should be limitable' do queue.limit = 1 expect(queue.acquire).to be expect(queue.acquire).not_to be end it 'should be resizable' do queue.limit = 0 expect(queue.acquire).not_to be queue.limit = nil expect(queue.acquire).to be end it 'should be countable' do queue.limit = 3 5.times { queue.acquire } expect(queue.probed).to eq 3 end it 'should be releasable' do queue.acquire expect(queue.probed).to eq 1 queue.release expect(queue.probed).to eq 0 end it 'should tell if paused' do expect(queue).not_to be_paused queue.pause expect(queue).to be_paused queue.unpause expect(queue).not_to be_paused end it 'should tell if blocking' do expect(queue).not_to be_blocking queue.block expect(queue).to be_blocking queue.unblock expect(queue).not_to be_blocking end it 'should be marked as changed' do queue = Sidekiq::Queue["uniq_#{name}"] expect(queue).not_to be_limit_changed queue.limit = 3 expect(queue).to be_limit_changed end end end end sidekiq-limit-fetch-3.4.0/spec/spec_helper.rb0000644000175000017500000000145513043067544020161 0ustar pravipravirequire 'sidekiq/limit_fetch' Sidekiq.logger = nil Sidekiq.redis = { namespace: ENV['namespace'] } RSpec.configure do |config| config.order = :random config.disable_monkey_patching! config.raise_errors_for_deprecations! config.before do Sidekiq::Queue.reset_instances! Sidekiq.redis do |it| clean_redis = ->(queue) do it.pipelined do it.del "limit_fetch:limit:#{queue}" it.del "limit_fetch:process_limit:#{queue}" it.del "limit_fetch:busy:#{queue}" it.del "limit_fetch:probed:#{queue}" it.del "limit_fetch:pause:#{queue}" it.del "limit_fetch:block:#{queue}" end end clean_redis.call(name) if defined?(name) queues.each(&clean_redis) if defined?(queues) and queues.is_a? Array end end end