I Replaced 50+ PHP Workers With One Go Process

Sentinel still runs on Laravel. I just stopped running Laravel in every monitoring region.

Until recently, each region was basically another copy of the application: Laravel, Horizon, Redis, nginx, PHP-FPM, supervisord and a pool of 50 to 60-something workers on a 16 GB server, using around 4 GB of RAM. The same region now runs as one Go process using about 25 MB on a 1 GB box, and it holds a tighter 30-second cadence than the PHP fleet ever did.

The old setup worked, and after a lot of tuning it worked pretty well. But I want Sentinel checking from a lot more regions around the world, and duplicating that setup for every new region wasn't going to make much sense.

So I started looking at alternatives.

The old setup

A monitoring region has a fairly simple job. It gets its monitors, runs their checks on schedule and reports what happened. The infrastructure I was using to do that wasn't nearly as simple. Each region ran on a 4 vCPU, 16 GB server with the full Laravel stack and somewhere around 50 to 60 Horizon workers. The boxes generally used around 4 GB of RAM.

The harder problem was keeping a tight cadence. A network check spends most of its time waiting on DNS, a connection, TLS, the response itself or eventually a timeout. With Horizon, a check that's waiting is still occupying a worker. I spent quite a bit of time tuning worker counts, queues, timeouts and scheduling to get the PHP nodes to reliably stay near a 30-second cadence. Even then, I'd occasionally see checks drift into the 40s under load.

That was manageable with a few regions. It was less appealing when I started thinking about putting Sentinel in a lot more of them.

Why I went with Go

I started looking at other ways to run the probes, and both Go and Rust were on the list. I ended up going with Go for a few reasons, but one of the more important ones was something specific to monitoring: the TLS fingerprint.

The existing PHP edges presented an OpenSSL-shaped ClientHello. If I replaced them with probes that suddenly looked like a completely different client to Cloudflare, Akamai and other WAFs, the results wouldn't really be equivalent anymore. Go has uTLS, which let me preserve that behavior. That matters when the whole point is to change the implementation without changing what the target sees.

Go's concurrency model also fits this workload really well. Hundreds of checks in flight can be hundreds of goroutines behind a semaphore. Rust and Tokio could certainly handle it, but I didn't need the extra complexity for a relatively small amount of networking code. Go's standard library covered most of what I needed, cross-compilation was straightforward, and deployment is a single binary. The first working node went from an empty repo to checking real monitors in an afternoon.

Rust might have produced a somewhat smaller binary or shaved off a little more memory, but neither matters here. The Go process is already tiny and barely touches the CPU. That's not where the next bottleneck is going to be.

Go measures. Laravel decides.

The most important architectural decision was how much responsibility to give the probe. Sentinel already handles HTTP, SSL, DNS, domain, ping and port checks, along with redirects, custom headers, authentication, keyword checks, JSON assertions and some more specialized checks. I didn't want to maintain a second implementation of all the logic that decides whether a monitor is healthy.

So the probe doesn't decide that. It performs the network operation and sends its observations back to Laravel. For an HTTP check, that can include the status code, DNS time, connection time, TLS time, TTFB and the response body when it's needed.

Laravel still handles accepted status codes, keyword and JSON assertions, certificate expiration, confirmation and recovery logic, regional consensus, incidents and notifications.

That split worked particularly well because Sentinel already separated performing a check from recording its result. Probe results go through the same path. Trimmed to two of the check types, the recorder looks like this:

class ProbeResultRecorder
{
    public function __construct(
        private readonly RedisResponseTimeService $redis
    ) {}

    public function record(
        Monitor $monitor,
        string $region,
        array $result,
        CarbonInterface $checkedAt,
        ?RegionTimestampBatch $timestampBatch = null
    ): void {
        (new MonitorCheck($monitor, $region))
            ->replay($result, $checkedAt, $this->redis, $timestampBatch);
    }

    public function recordSsl(
        Monitor $monitor,
        string $region,
        array $facts
    ): void {
        (new SSLMonitor($monitor, $region))->replay($facts);
    }
}

That kept the Go side relatively small and gave me a straightforward way to test it. I can run the PHP and Go implementations against the same targets and compare what they observed, while PHP remains the reference for what those observations actually mean.

How the probes get their work

Each probe authenticates with the control plane and periodically fetches a manifest. A trimmed HTTP monitor looks something like this:

{
  "id": 2481,
  "type": "http",
  "url": "https://api.acme.dev",
  "interval_seconds": 30,
  "timeout": 30,
  "method": "GET",
  "needs_body": false
}

The manifest comes from the same eligibility rules the existing dispatcher uses. If a monitor is paused, it disappears from the manifest. The same happens when an account is suspended or the regional configuration changes. The manifest uses an ETag, so most syncs are just a 304 rather than rebuilding and transferring the same data every time.

Results go back to the control plane in batches. If a probe can't reach it, the results are written to disk and sent later with their original timestamps. Probe releases are independent from Laravel releases as well. Each region is pinned to a probe version and the node periodically checks whether it should update, so deploying the main application no longer means restarting monitoring workers in every region.

The numbers

I've been able to run both architectures side by side, which made the difference pretty easy to measure.

The memory numbers make for a nice headline, but 16 GB versus 1 GB isn't really a fair comparison. The PHP servers are overprovisioned and could run on less. What I find more interesting is the cadence.

Getting the PHP workers to stay as close as they do to 30 seconds took a lot of work. The Go probes are holding around 30 to 31 seconds on much smaller machines without much effort, and there's still plenty of headroom. Based on what I'm seeing so far, they could comfortably run at a 15-second cadence. CPU and memory aren't close to being the limiting factors.

That's important because saving RAM wasn't really the goal. I want to add more regions and run more checks without having the infrastructure grow at the same rate.

Adding a region is a lot simpler now

With the old architecture, another monitoring location meant another Laravel stack and another worker fleet. Now it's one small binary on a small server. There's no application database at the edge and no Horizon cluster to manage. The probe gets its manifest, runs its checks and reports the results.

That changes the economics of expanding the network quite a bit. I can add geographic coverage without duplicating the application infrastructure every time, and there's enough headroom to tighten check intervals without solving that problem by throwing more PHP workers at it.

For Sentinel, that's the real benefit of the change. I wanted more regions, and the regional infrastructure was going to become increasingly expensive and cumbersome as I added them. The probes make that expansion much simpler.

A few things went wrong

IPv6 got me early. One SSL check was dialing by hostname and Go selected IPv6, while the HTTP path on the same node was using a pinned IPv4 address. A Cloudflare-fronted site behaved differently between the two paths and produced a false SSL incident. Every check type on a node now uses the same pinned address.

Cloudflare caused another problem when I brought up a new region. SentinelBot is verified and publishes its monitoring IPs, but a new address isn't necessarily recognized immediately. The probe started getting 403s from sites using managed challenges until Cloudflare caught up. New IPs now get staged before a region goes live.

The other thing I'm watching is the control plane. Making the probes much lighter means the regional nodes aren't the obvious scaling constraint anymore. Laravel still has to ingest and process every observation they send back. I've already cut some unnecessary work from that path, and the next change is separating acknowledgement from replay so receiving a batch isn't tied to processing every result during the same HTTP request.

At this point, the next limit is much more likely to be the control plane than the probes.

Where it ended up

I'm moving the existing regions over one at a time. Laravel isn't going anywhere. It still owns the application, accounts, monitor configuration, multi-region consensus, incidents, notifications and the logic that determines whether something is actually down. Go handles the part that spends most of its life waiting on the network.

The result is a much smaller regional footprint, tighter scheduling and a lot more room to expand the monitoring network. I went looking for a better way to run the edges, and a single Go process turned out to fit that job a lot better than a fleet of PHP workers.