I Replaced 50+ Horizon Workers With Go. Then Laravel Became the Bottleneck.

A few weeks ago I wrote about replacing Sentinel's PHP monitoring workers with one Go process per region. I ended that post with a guess. Once the regional nodes were cheap and barely loaded, the next limit was going to be the control plane. The Laravel app that hands every node its work and takes every result back.

It wasn't much of a guess. The control plane had been complaining since the fifth region.

The last PHP edge was retired on October 19. By October 21 the network had gone from six regions to fifteen, on six continents, each one a $5 box running a 50 MB process. Nothing on those boxes broke. Everything that broke was in Laravel. Some of it had shown up earlier in October, when the first two Go regions joined the four PHP ones, and going to fifteen made the rest impossible to put off. None of it was because Laravel was the weak part. Work that used to be spread across a fleet of 16 GB servers had been moved into one application on one box, and that application had to be written as if it mattered. Every fix was ordinary Laravel done a little more carefully than before. This post is about those fixes. Here's the short version.

What fifteen regions actually send

A probe node syncs its manifest every ten seconds and posts results in batches of up to 200, every five seconds. Each region's manifest is its HTTP monitors plus the SSL, DNS and domain sub-checks that region owns, and on the fastest tier a monitor is checked every 15 seconds from every region.

In the PHP days each region recorded its own results into the shared database from its own Horizon workers. Now one app on one server receives all of it over HTTP, and the question of what happens inside that request turned out to matter more than anything on the nodes.

Acknowledge first, record later

The first version of the results endpoint did the obvious thing. Validate the batch, loop through it, replay each observation through the same code path the PHP workers had used, return the per-item outcome. With two pilot regions it was fine. With every region on probes and nine more about to go live, a full batch was 200 results, and the request did 200 rounds of consensus, incident and timestamp work before the node heard anything back.

The node's behaviour made that worse. It spools results to disk and only drops what the control plane acknowledges, which is the right design for a flaky link, but it also means a slow acknowledgement holds the next batch. Pending results on a node is the number I watch, and it was climbing on a healthy network.

So the endpoint stopped recording. It now validates the batch, writes the payload to a file, inserts one row of metadata, and acknowledges every index. If the write fails nothing is acknowledged and the node keeps spooling. A full batch of 200 results validates in 0.47 seconds, and the write behind it is one file and one row. Recording moved to a queued job:

class DrainProbeNode implements ShouldQueue
{
    public const QUEUE = 'probe-ingest';

    public const BUDGET_SECONDS = 90;

    public int $tries = 1;

    public function middleware(): array
    {
        return [
            (new WithoutOverlapping("probe-drain:{$this->nodeId}"))
                ->dontRelease()
                ->expireAfter(self::BUDGET_SECONDS + 60),
        ];
    }

    public function handle(ProbeBatchProcessor $processor): void
    {
        $node = ProbeNode::find($this->nodeId);
        $deadline = now()->addSeconds(self::BUDGET_SECONDS);

        while (now()->lt($deadline)) {
            $batch = ProbeResultBatch::pendingFor($node->id)
                ->where('attempts', '<', ProbeResultBatch::MAX_ATTEMPTS)
                ->first();

            if (! $batch) {
                return;
            }

            if (! $this->drain($node, $batch, $processor)) {
                return;
            }
        }

        // Out of budget with work left: a fresh job picks it up, so one
        // node's backlog never pins a worker for longer than the budget.
        self::dispatch($this->nodeId);
    }
}

It's worth being precise about what the acknowledgement now means, because it changed. Before, a 200 from the results endpoint meant "recorded". Now it means "durably received". The payload is on the primary's disk and a row points at it. Recording happens later and can fail, so the guarantee the node gets is weaker, and the control plane takes on the job of never losing what it has acknowledged. The rest of the design is about keeping that promise.

WithoutOverlapping is keyed per node, because a node's results have to be recorded in the order they happened. A recovery recorded before the failure it follows is a phantom incident. dontRelease() means a second dispatch for the same node is dropped rather than retried while one is running, which is safe only because the running drain loops until the node has nothing pending and a scheduled command re-dispatches every minute for any node with batches still waiting. The lock expires a minute after the budget, so a worker that dies mid-drain can't hold a node hostage.

Each batch row keeps a cursor. If the processor can't reach storage partway through, it stops and the cursor is saved at that index, and the next attempt resumes there. Everything before the cursor was recorded and isn't recorded again. That makes recording at-least-once per batch, not exactly-once. A worker killed outright between recording a result and saving the cursor replays that slice, and nothing in the recorder refuses a result it has already seen. I've chosen to live with that, because the window is one batch after an abnormal kill and a duplicated sample is far cheaper than a lost one. If a batch throws three times it's abandoned with the error on its row. That is the one path where acknowledged results are lost, it's logged as an error, and I'd rather lose one batch loudly than block every batch behind it.

The queue gets its own Horizon supervisor with balance => false, sized so a backlog on one node never starves the others. The every-minute sweeper is the safety net, not the mechanism. On a normal day it finds nothing, and it prunes rows older than an hour on the way out.

The probe itself needed no change for any of this. It already spooled and already dropped only acknowledged items. The missing half was always on the Laravel side.

The 58 GB I didn't mean to store

The first version of that batch row kept the payload in a JSON column. It was the easy thing to do, and the payload can include captured page bodies up to 2 MB each for failed checks, because the control plane is what diagnoses them.

Twenty hours later the primary's root disk had gone from 24% to 48%.

The batch table was 837 MB. The hourly prune was deleting rows on schedule. The disk was being eaten by MySQL's binary log. Every batch insert, and every delete an hour later, went through the binlog and across the replication stream to the standby, and the binlog keeps everything for 30 days by default. It had reached 750 files and 75 GB, about 58 GB of that in those twenty hours, for data whose useful life was about five seconds.

Payloads are files now, under the app's private storage, written by the endpoint and deleted by the drain job the moment a batch is recorded. The row holds the path, the count, the cursor and the outcome. A sweeper removes any orphan file older than a day. The binlog expiry on the primary is seven days, which is a change I should have made long before this. Purging the old logs took the disk from 48% to 13%.

The rule I took from it is short. Nothing that exists per check result goes in MySQL unless it needs to replicate. Laravel makes it very easy to put a JSON column on a model and move on, and for most data that's fine. For data that arrives in a steady stream and dies a few seconds later, the database isn't a buffer.

An ETag that isn't a hash

Every node asks the control plane for its manifest every ten seconds. The point of polling that often is that a new monitor should be checked within seconds of being created, not minutes. The cost is 15 regions times six polls a minute, each one asking "anything new?"

The first implementation answered that by building the manifest and hashing it, so the ETag was an honest content hash. Correct, and a full manifest build per poll. At fifteen regions that would have been several thousand manifest builds an hour to say "no" almost every time. I replaced it at five.

The ETag is now a version stamp that lives in the cache:

final class ProbeManifestVersion
{
    public const SCHEMA = 2;

    public static function current(): string
    {
        $epoch = Cache::rememberForever('probe:manifest:epoch', fn () => (string) now()->getTimestamp());
        $counter = (int) Cache::get('probe:manifest:counter', 0);

        return '"'.$epoch.'.'.$counter.'.s'.self::SCHEMA.'"';
    }

    public static function bump(): void
    {
        try {
            if (! Cache::has('probe:manifest:counter')) {
                Cache::forever('probe:manifest:counter', 0);
            }

            Cache::increment('probe:manifest:counter');
        } catch (Throwable $e) {
            report($e);
        }
    }
}

A poll that presents the current stamp is a 304 for the price of two cache reads. The counter moves from Eloquent model events, and which events turned out to be the interesting part:

static::created(fn () => ProbeManifestVersion::bump());

static::updated(function (Monitor $monitor): void {
    if ($monitor->wasChanged(ProbeManifestVersion::MONITOR_ATTRIBUTES)) {
        ProbeManifestVersion::bump();
    }
});

static::deleted(fn () => ProbeManifestVersion::bump());

A monitor is saved every time it's checked, because its status and last-checked time live on the row. If every save bumped the version, every node would refetch every manifest a few times a second and the whole idea would collapse. So the updated hook only bumps when an attribute that reaches a manifest row changed, and created is a separate hook rather than a check on wasRecentlyCreated inside saved, because that flag stays true on the creating instance for the rest of the request. I learned that one the slow way.

Two more things live in that stamp. The epoch is set on the first read after a cache flush, so a flush can't hand a node a counter value it already holds. And the schema number is part of the tag, so when the manifest gains a field, every node holding a cached copy refetches at once instead of waiting for the next monitor change. The node's own version is appended too, after an upgraded node was answered 304 forever against a manifest shape it had never seen.

The effect was the one I wanted. A monitor created in the dashboard is checked from every region in about ten seconds.

It also produced the one real outage of the migration, four minutes long. A node that's installed but not yet active syncs an empty manifest so I can watch it on the network before it carries work. When Seattle flipped to active, the node presented the ETag it had cached against the empty list, the stamp hadn't moved, and it was answered 304 with no monitors for four minutes until I bumped the version by hand. An inactive manifest now carries its own fixed tag.

Things that were fine at four regions

None of these were design problems. They were limits that sat comfortably out of reach until the fifth or the tenth region arrived.

Reverb's 10 KB message cap

The admin dashboard gets an infrastructure health snapshot every minute over a private channel. The payload was one event with every region's vitals in it. The fifth region pushed it past 10 KB, and 460 broadcast jobs failed with "Payload too large" overnight before I looked. It's one event per region now, and the page merges them. The incident event had hit the same cap a few weeks earlier when a long incident's diagnostics outgrew it, so that one already sent a count instead of the activity log. I should have gone looking for the next payload that grows with the network.

Hydrating a hundred thousand models

The monitor page draws a seven-day percentile series. The query had always loaded the raw checks for the window as Eloquent models and bucketed them in PHP. A 30-second monitor on four regions fit in the 128 MB request limit. The fifth region tipped it over, and the chart became a 500. The fix is the one you'd expect:

$rows = $monitor->responseTimes()
    ->toBase()
    ->whereIn('region', $regions)
    ->where('checked_at', '>=', $from)
    ->selectRaw('(FLOOR(UNIX_TIMESTAMP(checked_at) / ?) * ?) AS bucket, response_time', [$bucketSeconds, $bucketSeconds])
    ->cursor();

foreach ($rows as $row) {
    $buckets[(int) $row->bucket][] = (int) $row->response_time;
}

Plain integers, bucket computed in SQL, no model and no Carbon instance per row. Each section of that page now runs within a few megabytes of baseline. The general lesson is that anything on a page that scales with regions times checks has to stream, and toBase()->cursor() is the cheapest way to get there.

Health queries that multiplied by region

The dashboard has a strip showing the health of every region. It was computed on the request, with one query per region and one Redis read per monitor per region. At four regions nobody noticed. By late October the dashboard request took 4.6 seconds, 2.8 of them in SQL.

The strip is now computed once a minute by the job that already broadcasts it, and cached, so a page request only reads it. The computation is one grouped query and one pipelined Redis pass, whatever the region count. The same pass found an index that ended in the wrong column for the query using it, which is the sort of thing a slow page is good for.

nginx's body limit

Sydney's node stopped posting half an hour after it went live, while its sync calls stayed healthy. Its spool kept climbing. The node's journal said 413 on every post. The server tooling in front of the app had set client_max_body_size to 2 MB, and a batch with a couple of captured bodies in it was 4 MB. The limit went to 16 MB, and the probe learned to halve its batch on a 413 down to a single result, then drop that one with a log line rather than retry it forever. Head-of-line blocking on a retry loop is a quiet way to lose a region.

What came next wasn't on my list

The first draft of this post ended with a prediction. Every node was reading zero pending batches, the probe-ingest supervisor was idle, and I guessed the next limit would be the hourly response time aggregator, which re-rolls 36 hours of data every run.

The next two problems arrived within a week, and neither was about throughput.

On November 7 the Seoul node lost its network for fourteen hours. The probe did what it was built to do. It kept checking from its cached manifest, failed every check at DNS, spooled the results and replayed all of them when the link came back. The control plane did what it was built to do as well. It acknowledged every batch and recorded every result. No incident opened and nobody was alerted, because no other region agreed. But every monitor's Seoul history now held fourteen hours of failures that never happened.

Ack-first had made the control plane very good at not losing data, and nothing in it asked whether the data was true. One result can't tell a broken domain from a node with a dead resolver. A batch can. A batch that arrives long after its checks ran, and is almost entirely network failures across unrelated hosts, is describing the node. The drain now drops those failures and logs that it did. A live node reporting a real outage uploads within seconds, so it's always recorded.

Two days later the path through Cloudflare from the far regions slowed to a crawl. My server sat idle while uploads waited. A 1 MB post from Johannesburg took 53.5 seconds through Cloudflare and 2.2 seconds straight to the server. Results piled up on healthy nodes for over an hour, and the number I watch was reporting on a network I don't run. Probes now post results to a hostname that skips the proxy and serves nothing but the probe API, and they fall back to the main one if it fails. The sync response tells each node where to post in a header, not a manifest field, so changing it never moves the ETag.

So I've stopped predicting the next limit. The number I watch is still pending results per node. When it stops reading zero on a quiet day, something between the probe and the database is behind, and most of that distance is Laravel.

I don't think any of this is remarkable code. A queued job with a per-key lock, a cache counter as a version stamp, a file instead of a column, a cursor instead of a collection. What changed is that the probes stopped hiding the cost of the control plane behind their own cost. When each region was a 16 GB server running 60 workers, Laravel's share of the work was spread across the fleet and nothing on the primary was ever the bottleneck. Fifteen tiny nodes talking to one app moved all of it into one place, and that place had to get a bit more disciplined.

It did, and it's still Laravel.