Lowkiq is the new server for streamlined background task processing for ruby and redis. It was created at BIA-Technologies, a company developing logistics solutions.
In this article I will talk about the problems of processing background jobs that we encountered, and how to solve them.
ElasticSearch . 2 : A B. , , . , A . , , .
, . . ( ):
Sidekiq . redis
, , .
. redis, , . , , . , .
, "" . -. , , — .
Kafka. Kafka , .
. Lowkiq.
, A . . redis.
. , .
redis. :
:
- , score
- perform_in
- retry_count
score perform_in — unix timestamp.
, , .
, payloads . , .
, , , perform_in. perform_in, Lowkiq , - .
Lowkiq inherited sharding from the indexer, but got a more flexible system for distributing shards between threads. Now each thread can process several shards, which allows you to add new queues without increasing the number of threads.
Lowkiq, like the indexer, ensures that no other handler can process a task with the same identifier in parallel.
Several of our projects use Lowkiq in production throughout the year. However, we still use Sidekiq and other queuing systems in normal cases. If you are facing the same problems as us, try Lowkiq .