Image

The Power of RabbitMQ

During Drupal development, very often we are faced with a situation where we have large tasks that need to be broken up and processed at a more manageable pace. Naturally, we turn to the native Drupal Queue (introduced in Drupal 7). However, the Drupal Queue has some major limitations when it comes to speed and reliability which can make it an unviable option.

Speed

• The Drupal Queue is processed by cron and is therefore bound by its minimum run time interval of once a minute.
• The Drupal Queue doesn’t natively support multiple consumers/workers. On each cron run, Drupal loops through the queue processing items (until it hits the maximum time limit) and provides no way of processing items in tandem.

Reliability

• When each hook_cron_queue_info is implemented, it deletes only fully processed items from the queue. If an item in the queue hits a PHP error or timeout, it does not actually release the item for reprocessing.

Here is where the power of RabbitMQ comes into place. RabbitMQ is a third party messaging service which can act as a great enhancement to the Drupal Queue and is supported with Drupal module.

RabbitMQ solves all of our speed and reliability issues:

• Supports multiple consumers - the sky is the limit here or better stated, your server power. You can run as many consumers in tandem as you want, adding great speed to the processing time.
• Consumers can be stored on multiple servers - RabbitMQ runs on a single server and consumers can exist on separate servers to help distribute the load.
• Provides message acknowledgments - if an item claimed by a consumer fails to process, it is then put back into the queue for processing. RabbitMQ maintains this by only fully removing an item from the queue when it receives a notice that it was successfully processed. This allows the queue to keep moving while ensuring that there is no data loss.
• RabbitMQ’s capabilities can be augmented by the use of RabbitMQ cli consumer - This program is written in Go which is much better at running long-running processes. RabbitMQ cli consumer constantly watches for incoming signals from the queue and processes them almost instantaneously.