Message Queues

Have you ever had a task that takes forever? Maybe you’ve let the dishes pile up for a little too long. Now that you’re keen to do some cooking, you’ve found that you have no clean pots or plates. You’re blocked from creating tasty dishes until you take care of the washing up.

That’s exactly why message queues exist. Well, I suppose not exactly. Look, let’s have a closer look at the problem with a more realistic scenario. Please note that this post aims to explain queueing as a concept, rather than any specific software solution. Still happy? Okay! Read on.

Consider PHP, a language that has progressed by leaps and bounds in recent years in terms of both flexibility and performance. However, there are some limitations that even the most recent versions of PHP cannot fix.

Let’s consider concurrency. I know, I know, I can hear you gasp from here. PHP isn’t a great language for threading, or for asynchronous code in general, and you’re quite right. I’m actually talking about a different type of concurrency.

Let’s examine how PHP and Nginx work together for a moment.

We’ll simplify things a little, but essentially, when Nginx receives a new request for a PHP file, it will create a new PHP-FPM process that will handle the script, and pass the processed response back to Nginx. The web-server can spawn a number of these PHP-FPM processes in parallel. In fact, if you look at the process list of a web-server serving a popular PHP website, you’ll likely see a number of entries for PHP-FPM.

While Nginx will happily spawn a large number of PHP-FPM processes to handle load, there is a configurable limit. Once that limit has been hit, the web-server won’t serve the content as intended. In fact, Nginx also has a configurable limit of simultaneous connections that it can handle. So there are two opportunities to run out of resources when serving web content.

“But Dayle,” you say, “I didn’t come to your blog to learn about DevOps.” That’s perfectly fine, but the bit that’s really important to grasp is that the web-server cannot serve an infinite number of users at once. This is why PHP applications work best when they do their work quickly and serve the response as soon as possible. Freeing up resources (the fpm pool) for new requests.

For now, let’s imagine that FPM can only handle three requests simultaneously. That’s a rather small number compared to the general configuration, but please, bear with me.

User one visits the site, and our script loads some content from an external host. A third party API, shall we say? The API is slow, and the user watches a loading spinner in the corner of the browser.

While they waiting for their request to finish, that means that there are only 2 processes now available to process requests. If two others were to hit the same endpoint, the web-server wouldn’t be able to handle any further requests. That’s a big problem! We don’t want angry users.

In situations such as these, ones where a process may take a long time to complete, we can consider using a queue. However, before we do so, we must ensure that our current request need not send an immediate response.

For example, let’s say we click a button on a website, and it should send an email. Who knows how long it will take to send the email, and we don’t really know when it will be received. So why don’t we queue it instead?

The flow will go like this:

  • User: Clicks the button.
  • App: Queues an email to be sent. Sends a response immediately.
  • User: See’s “Thanks, your email has been sent.”

Because we queue the item (a fast process) rather than send the email directly, the user isn’t made to wait for confirmation. The page reloads quickly.

So let's think about queueing for the moment? Essentially, we’re storing some work to act upon later. It’s performing an action with some data. That sounds a lot like a function to me, don’t you agree? A function has a name and some parameters. So that’s what our queued message will look like. Here’s an example:

{
    "name": "email.send",
    "data": {
        "user": 123,
        "email": "button_push"
    }
}

The message above has a name, or type ‘email.send’ and a data payload containing the ID of the user to send the email to, and which email to send.

We could have added subject line, entire email content and much more, but that wouldn't be ideal. Queued messages work best with the minimal data set required to represent the work to be done.

For example, we can use the parameters above to retrieve a user from the database by ID, and then we will have their email address. We could also easily create a system to locate and load the email template, content, and subject line from the email type.

Once we have our queue message, we can place it on our queue. You can think of a queue as a stack. At first, the stack will contain no items.

  • (Empty)

We can place as many items on the queue as we like, here we’ve added a few send email messages to the queue. I’ll leave the JSON out for this example. Format doesn’t really matter, it’s the data that’s important.

  • Send Email: user:54 email:welcome
  • Send Email: user:67 email:welcome
  • Send Email: user:142 email:welcome
  • Send Email: user:54 email:password_reset

So this queue, where is it stored? Good question. I mean, those just look like records to me, don’t you agree? I suppose we could use the database? Sure, why not! We’ll insert each new message into a database table containing a field for the message name, and another for the message parameters. That’s simple stuff, isn’t it?

And now we’re done! Oh wait, I must be mistaken. Now we’re just throwing messages at the database, but we’re no closer to sending our emails. They’ll just sit there forever, stacking and stacking until we fill the hard disk of our database server!

What we need next, is something called a ‘consumer’ or ‘worker’. It’s a special application that runs independently from your website, and churns through the queue, tackling each message one by one. Here’s how it might look in some basic pseudo code.

while (true) {  
    if (message = queue.getMessage()) {
        // Get user from database.
        // Fetch template and render it.
        // Send email using SMTP.
        message.delete();
    }
}

Hah! Stupid Dayle, can’t you see that you’ve created an endless loop! That’s a rookie coding mistake!

You’re absolutely right! That is an endless loop, and it’s exactly what we want. You see, the consumer will always be running, checking the queue (polling) for new messages to process. Queue consumers are normally daemonized, a process that runs endlessly. Even once it has emptied the queue of messages, and all emails have been sent, it will sit there eagerly asking the queue for more tasks to complete.

You could set the consumer to handle events of multiple names in different ways. For example, a consumer might know how to send emails, deliver authentication text messages, or capture monthly subscriptions. It might be able to handle all of the above events. It’s really up to you.

What’s important to remember, is you can’t be sure when a message will be processed. You could make an assumption based on the length of the queue, but sometimes it’s hard to say how long jobs will take to be processed. That means that queues are best for non-urgent tasks that ‘will be done as soon as possible, but perhaps not right this second’.

Let’s consider the traits of an action that’s ideal to be queued, once again.

  • May be resource intensive or long running.
  • Does not need to provide resulting data immediately.
  • Does not need a predictable processing time.

Normally, queues are ideal for background business processes and communications tasks but can be suited to more. Here are a few things that I’ve used queues for.

  • Sending emails or text messages.
  • Logging content to one/many places.
  • Taking payments.
  • Generating (warming) cache content.
  • Updating the search index for some content.

In this example, we’ve used a database table to store our queue because it’s a data storage method that’s very familiar to most developers, however, there are a number of software solutions that are much more effective.

Not only will these solutions likely be faster than using the database, but they have a feature set tailored to queuing jobs. For example, supporting locking (ensure a job doesn’t get processed twice) fanout (deliver a message to multiple queues) and more!

That’s all there is to queues as a concept, so feel free to escape at this point. However, I have a few bonus topics for those who are naturally curious.

Performance & Parallelism

If you find yourself in the position where the number of messages in your queue are rising faster than your consumers are processing them, then don’t panic! It means you're busy, maybe collecting loads of money. It’s a good problem to have!

But you’re probably looking for a solution, right? Well, it’s actually a really simple one! Simply run more consumers. If you run more consumers in parallel, you’ll be able to double..triple..etc the processing time of your jobs. Run as many parallel consumer instances as you require to keep the message numbers down.

Companies with advanced infrastructure will use DevOps techniques that scale the number of consumers dynamically depending on the number of messages in the queue. This can be an extremely cost-effective way of managing queues.

You can also consider splitting your consumers by type. Arranging so that some consumers handle different jobs to others. Many developers opt for having high/low urgency queues so that only urgent tasks will be placed on the high queue, keeping the number of messages (and thus, response time) low.

Useful Frameworks

Many frameworks come with a solution for queueing, and you’ll also find a wide number of open-source consumers online available for the different queuing mechanisms.

Most frameworks treat queuing in a similar manner to routing. Instead of using the URL to route requests, we’ll use the message ‘name’ or ‘key’ to select the appropriate code to handle the job.

Why not take a look at how Laravel handles queued tasks?

Gotchas

A lot of the gotchas relating to queues are related to asynchronous programming in general. It’s impossible to judge the amount of time between a message being placed on the queue, and it being processed. A lot can happen in the interval.

Consider the following situation. We place a message on the queue to send an email to user 32, embedding their email ‘foo@example.com’ within the message. Between the message being placed on the queue, and it being processed by the consumer, the user may have changed their email address on the website. However, the queued message will still contain the old email address, and the email will be sent to the wrong destination.

For this reason, it’s best to only store stable information on the queue, in the above example, the primary key (id) field of the user table might be a better choice. We can use that to retrieve the email address from within the consumer, and it’s very unlikely that a user is able to change their id.

Depending on the queueing solution that you end up with, you might find race conditions within your consumers. For example, two consumers may take the same job from the queue at the same time, if both were to process it, it might cause odd problems. An email might be sent twice, for example.

To avoid these issues, it’s best to implement a locking mechanism, or simply to use a trusted message queue solution that will no doubt handle this for you.


Well, I hope you’ve learned something new from the article! If there’s anything you find confusing, please don’t be afraid to ask questions in the comment. There’s no such thing as a stupid question!

Do you have another programming topic that you'd like explained in my own special way? Feel free to make a request in the comments!

Have a wonderful day!