Get best of the week

Making our product ready to scale with Laravel queues

A translation of the article was prepared especially for students of the “Framework Laravel” course .



Hi, I'm Valerio, a software engineer from Italy.

This guide is intended for all PHP developers who already have online applications with real users, but who lack a deeper understanding of how to implement (or significantly improve) scalability in their system using Laravel queues. I first learned about Laravel at the end of 2013 at the start of the 5th version of the framework. Then I was not yet a developer involved in serious projects, and one of the aspects of modern frameworks, especially in Laravel, which seemed to me the most incomprehensible, were Queues.

Reading the documentation, I guessed about the potential, but without real development experience, all this remained only at the level of theories in my head.
Today, I am the creator Inspector.dev- realtime dashboard , which performs thousands of tasks every hour, and I understand this architecture much better than before.



In this article I am going to tell you how I discovered queues and tasks and what configurations helped me process large amounts of data in real time, while saving server resources.

Introduction


When a PHP application receives an incoming http request, our code is executed sequentially step by step until the request is completed and a response is returned to the client (for example, the user's browser). This synchronous behavior is truly intuitive, predictable, and easy to understand. I send an http request to the endpoint, the application extracts data from the database, converts it to the appropriate format, performs some additional tasks and sends them back. Everything happens linearly.

Tasks and queues introduce asynchronous behavior that violates this line flow. That's why these features seemed a little strange to me at first.
But sometimes an incoming http request can trigger a cycle of time-consuming tasks, for example, sending email notifications to all team members. This may mean sending six or ten emails, which may take four or five seconds. Therefore, every time the user clicks on the corresponding button, he needs to wait five seconds before he can continue to use the application.

The more the application grows, the worse this problem becomes.

What is a task?


A job is a class that implements the handle method that contains the logic that we want to execute asynchronously.

<?php

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Bus\Queueable;


class CallExternalAPI implements ShouldQueue
{
    use Dispatchable,
        InteractsWithQueue,
        Queueable;
        
    /**
     * @var string
     */
    protected $url;

    /**
     *    .
     *
     * @param array $Data
     */
    public function __construct($url)
    {
        $this->url = $url;
    }
    

    /**
     *  ,   .
     *
     * @return void
     * @throws \Throwable
     */
    public function handle()
    {
        file_get_contents($this->url);
    }
}

As mentioned above, the main reason for concluding a piece of code in Job is to complete a time-consuming task, without forcing the user to wait for it to complete.

What do we mean by “laborious tasks”?


This is a natural question. Sending emails is the most common example used in articles that talk about queues, but I want to tell you about the real experience of what I needed to do.

As a product owner, it is very important for me to synchronize user travel information with our marketing and customer support tools. Thus, based on user actions, we update user information in various external software via API (or external http calls) for service and marketing purposes.
One of the most commonly used endpoints in my application can send 10 emails and make 3 http calls to external services before completion. No user will wait so much time - most likely, they all simply stop using my application.

Thanks to the queues, I can encapsulate all these tasks in the allocated classes, transfer the information necessary for the execution of their work to the constructor, and schedule them to be completed at a later date in the background so that my controller can immediately return the answer.

<?php

class ProjectController 
{
    public function store(Request $request)
    {
        $project = Project::create($request->all());
        
        //  NotifyMembers, TagUserActive, NotifyToProveSource 
        //   ,     
        Notification::queue(new NotifyMembers($project->owners));
        $this->dispatch(new TagUserAsActive($project->owners));
        $this->dispatch(new NotifyToProveSource($project->owners));
        
        return $project;
    }
}

I do not need to wait for the completion of all these processes before returning the answer; on the contrary, the wait will be equal to the time required to publish them in the queue. And that means the difference between 10 seconds and 10 milliseconds !!!

Who performs these tasks after sending them to the queue?


This is the classic publisher / consumer architecture . We have already published our tasks in the queue from the controller, now we are going to understand how the queue is used, and, finally, the tasks are performed.



To use the queue, we need to run one of the most popular artisan commands:

php artisan queue:work

As the documentation says:
Laravel includes a queue worker who processes new tasks when they are queued.

Great! Laravel provides a ready-made interface for queuing tasks and a ready-to-use command to retrieve tasks from the queue and execute their code in the background.

Supervisor Role


It was another “weird thing” in the beginning. I think it's normal to discover new things. After this stage of training, I write these articles to help myself organize my skills, and at the same time help other developers expand their knowledge.

If the task fails while throwing an exception, the team will queue:workstop its work.

In order for a process queue:workto run continuously (consuming your queues), you must use a process monitor such as Supervisor to ensure that the command queue:workdoes not stop working even if the task throws an exception. Supervisor restarts the command after it crashes, starting again from the next task, setting aside the throwing exception.

Tasks will run in the background on your server, no longer dependent on the HTTP request. This introduces some changes that I had to consider when implementing the task code.

Here are the most important ones that I will pay attention to:

How can I find out that the task code is not working?


When running in the background, you cannot immediately notice that your task is causing errors.
You will no longer have immediate feedback, such as when making an http request from your browser. If the task fails, it will do it silently and no one will notice.

Consider integrating a real-time monitoring tool such as the Inspector to surface every flaw.

You do not have an http request


There is no more HTTP request. Your code will be executed from cli.

If you need query parameters to complete your tasks, you need to pass them to the task constructor for later use at runtime:

<?php

//   

class TagUserJob implements ShouldQueue
{
    public $data;
    
    public function __construct(array $data)
    {
        $this->data = $data;
    }
}

//       

$this->dispatch(new TagUserJob($request->all()));

You don't know who logged in


There is no longer a session . In the same way, you will not know the identity of the user who is logged in, so if you need information about the user to complete the task, you need to pass the user object to the task constructor:

<?php

//   
class TagUserJob implements ShouldQueue
{
    public $user;
    
    public function __construct(User $user)
    {
        $this->user= $user;
    }
}

//       
$this->dispatch(new TagUserJob($request->user()));

Understand how to scale


Unfortunately, in many cases this is not enough. Using one line and a consumer may soon become useless.

Queues are FIFO buffers (“first in, first out”). If you have planned a lot of tasks, maybe even of different types, they need to wait for others to complete their tasks before completing.

There are two ways to scale:

Several consumers per queue.



Thus, five tasks will be removed from the queue at a time, speeding up the queue processing.

Special-Purpose Queues

You can also create specific queues for each type of task to be launched with a dedicated consumer for each queue.



Thus, each queue will be processed independently, without waiting for other types of tasks to complete.

Horizon


Laravel Horizon is a queue manager that gives you full control over how many queues you want to set up, and the ability to organize consumers, allowing developers to combine these two strategies and implement the one that suits your scalability needs.

The launch occurs through php artisan horizon instead of php artisan queue: work . This command scans your configuration file horizon.phpand starts a series of queue workers depending on the configuration:

<?php

'production' => [
    'supervisor-1' => [
        'connection' => "redis",
        'queue' => ['adveritisement', 'logs', 'phones'],
        'processes' => 9,
        'tries' => 3,
        'balance' => 'simple', //   simple, auto  null

    ]
]

In the above example, Horizon will start three queues with three processes assigned to process each queue. As mentioned in the Laravel documentation, Horizon's code-driven approach allows my configuration to remain in a version control system where my team can collaborate. This is the perfect solution using a CI tool.

To find out the meaning of configuration parameters in detail, read this wonderful article .

My own configuration



<?php

'production' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['default', 'ingest', 'notifications'],
        'balance' => 'auto',
        'processes' => 15,
        'tries' => 3,
    ],
]

Inspector mainly uses three queues:

  • ingest for processes for analyzing data from external applications;
  • notifications are used to immediately schedule notifications if an error is detected while receiving data;
  • default is used for other tasks that I do not want to mix with ingest and notifications processes.

Through the use of balance=autoHorizon, he understands that the maximum number of activated processes is 15, which will be distributed dynamically depending on the loading of queues.

If the queues are empty, Horizon supports one active process for each queue, which allows the consumer to immediately process the queue if a task is scheduled.

Concluding observations


Concurrent background execution can cause many other unpredictable errors, such as MySQL "Lock timed out" and many other design problems. Read more here .

I hope this article has helped you use queues and tasks with more confidence. If you want to know more about us, visit our website at www.inspector.dev

Previously published here www.inspector.dev/what-worked-for-me-using-laravel-queues-from-the-basics-to -horizon


Get on the course and get a discount.

More articles:


All Articles