Timeout and Memory Leak Issues in Horizon With Laravel Octane

TL;DR #

• Uncomment FlushUploadedFiles and CollectGarbage event listeners in Octane config.
• Check memory_limit in Horizon, PHP-FPM and PHP-CLI config and increase if necessary.
• Check Horizon timeout configs and increase if necessary.
• Check max_execution_time in Octane, PHP-FPM, and PHP-CLI config and increase if necessary.

Even though I was aware of what Octane does and the super-fast performance it offers for Laravel applications, this was the first time we used it in a recent project. From the Octane documentation and numerous YouTube videos, I knew there were special considerations to keep in mind when designing the architecture of the application while using Octane. Memory leaks were one of them.

In our application, we have heavy usage of WebSockets. After launching a second release, usage skyrocketed, and we started encountering thousands of failed events. I began investigating the issue, and the first error I found was this:

Illuminate\Broadcasting\BroadcastException: Pusher error: cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://******/apps/******/events?auth_key=******&auth_timestamp=1725733807&auth_version=1.0&body_md5=159784e0a20c9b122f08b6d7d7389678&auth_signature=a424a1859671b8af49b1d242edaf85ec37bf8cb23b08bda58b195b2bbc5ec260. in /******/forge/******/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:164

So, it was a timeout error; some long-running events were failing for some reason. I double-checked the horizon and queue configs for all the timeout values. Everything seemed fine. The stacktrace also gave no useful clue. We use Laravel Forge to maintain our apps, and we recently swtiched from Soketi to Laravel Reverb. I went into the server logs and checked the daemon logs for all services (Horizon, Octane, and Reverb). In the Reverb log, I found something interesting:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /******/forge/******/vendor/laravel/telescope/src/IncomingEntry.php on line 79

From my understanding, due to memory exhaustion, PHP was crashing along with Reverb, causing the Pusher connection to fail, which resulted in the timeout error. 536870912 bytes is around ~536 megabytes, and we had set the memory limit to 512MB in our PHP config. We had enough memory available, so we increased the limit significantly. For a while, everything seemed normal. However, later that day, memory was exhausted again.

To resolve the issue quickly, we increased the limit to 8GB for the moment. Pretty insane for a Laravel Application, I know. But when you serve an application using Octane, it’s expected to consume more memory than usual since everything is running in memory. Despite this, we suspected a potential memory leak when it still didn’t hold up for long. Even for an in-memory application, the consumption seemed unusually high based on our estimates. It was driving us crazy because it felt like the solution was right before our eyes, yet we were missing it.

I started looking into Octane’s config. Our application handles a lot of file uploads. Octane provides a variety of event listeners in its config file, but you need to enable them based on your requirements. First thing I enabled was FlushUploadedFiles for RequestTerminated event:

//...
'listeners' => [
    //...
    RequestTerminated::class => [
        FlushUploadedFiles::class,
    ],
    //...
],
//...

Another event listener I uncommented was CollectGarbage for OperationTerminated event. This one is very important.

//...
'listeners' => [
    //...
    OperationTerminated::class => [
        //...
        CollectGarbage::class,
    ],
    //...
],
//...

Along with that, I increased the Garbage Collection Threshold to 512MB:

//...
'garbage' => 512,
//...

After these changes, we saw a dramatic improvement in the memory leak situation. The memory usage dropped almost 50%. The funny thing is, I overlooked another very crucial detail right after the garbage collection settings.

For nearly a day, no more errors were reported. Unfortunately I started seeing failed events again due to the timeout issue. But this time, neither PHP nor Reverb was crashing; it was purely timeout issue. We spent days tweaking various settings, including increasing the max_execution_time in both PHP-FPM and PHP-CLI configs, but there was no noticeable improvement. So, I decided to go through the Octane config once more. To my disbelief, I found myself dumbfounded for missing this the first time:

//...
'max_execution_time' => 30,

This is the description of the settings:

The following setting configures the maximum execution time for requests being handled by Octane. You may set this value to 0 to indicate that there isn’t a specific time limit on Octane request execution time.

The value is in seconds. From our error message, we know that it was 30001 ms which is exactly 30s. I increased this to 120s.

Additionally, I changed default_socket_timeout to 120s from 60s in PHP config which I felt necessary for our use case. My colleague also adjusted some Reverb connection settings. So far, we haven’t faced anymore memory leak or timeout issue since then. 🥂