Case study — How I Traced a Googlebot-only HTTP 500 Across Next.js, Cloud Run and the Hosting Edge

This case study intentionally omits proprietary source code, internal repository information, cloud project identifiers, operational timestamps, verbatim vendor correspondence and confidential client data. What remains is the investigation method, the failure chain at a technical level, and the operational lessons.

Context

A production website for a Japanese enterprise client: Next.js on the App Router, running on Cloud Run, with public traffic served through Firebase Hosting.

One property of that setup shaped the entire investigation. The project team could observe the application and Cloud Run directly. The edge layer serving the public domain was largely outside that visibility — there was no direct access to inspect its full internal behaviour.

That gap is where the incident lived.

Problem

Search visibility dropped sharply over a short period. The website itself was working normally for real users.

That combination rules out the usual explanations immediately. It is not a traffic problem and not a content problem — it is an infrastructure problem specific to how crawlers are served.

Symptoms

Four properties made this hard to trace, and they are worth stating because each one defeats a standard debugging reflex:

The failure was invisible to users and to conventional monitoring. Browsers and other bots received HTTP 200. An ordinary uptime check saw nothing wrong.

The failure left no trace where everyone looks first. Googlebot's failing page requests never reached the origin, so Cloud Run logs contained no stack trace and no corresponding 5xx.

Cause and effect were separated in time. The application regression had shipped well before the incident surfaced. Correlating against the most recent deploy therefore pointed in the wrong direction.

It was a multi-layer failure chain, not a fault. Looking only at the application showed one broken response at /robots.txt. Looking only at SEO showed Googlebot had stopped crawling. The value was in connecting the two into a single chain of causation.

Investigation

The guiding principle throughout: each step had to eliminate a class of hypotheses, not merely accumulate evidence for whichever one currently looked most likely.

That matters more than it sounds. The initial hypothesis leaned heavily towards infrastructure, and gathering supporting evidence for it would have produced a confident, incomplete answer — the application regression would have been missed entirely.

Diagnostic experiments

1. A two-dimensional matrix to locate the failing layer

Rather than auditing configuration one setting at a time, I tested the request path against the user agent:

The site-wide 500 that Googlebot saw existed only at the intersection of public edge and Googlebot. That eliminated normal application page requests as the direct source — but it did not yet explain what condition triggered the edge behaviour.

2. Comparing several crawlers to find the exact boundary

AdsBot-Google is the informative row: also Google, also crawling, but unaffected. So this was not "bots are blocked" and not "the site is down". The behaviour tracked something about how requests carrying Googlebot in the user agent were handled on the public path.

3. Building a timeline from logs instead of guessing

Crawler activity ran steadily up to a point and then stopped absolutely, while other bots continued unchanged. A static snapshot would not have shown that. The sharp edge in the timeline indicated a state threshold had been crossed rather than a gradual degradation.

4. A control site

I ran the same tests against a different website on the same architecture. It showed no 500 behaviour for Googlebot, which eliminated "platform-wide outage" as an explanation.

5. From the edge symptom to the real dependency

With ordinary page requests ruled out, I moved to the dependencies Googlebot checks before crawling — /robots.txt first. That is where the chain became visible.

Root cause

An application regression and an amplification mechanism at the hosting edge. Neither half explains the incident alone; only together do they produce the full failure chain.

In the application. After the site was upgraded to a multi-language flow, unknown dot-paths such as /robots.txt no longer fell through to a plain 404. Part of the i18n and 404 logic used runtime-only request APIs inside a static render context, which turned what should have been a 404 into an HTTP 500. This was a genuine code regression, and I fixed it directly.

At the hosting edge. After escalating to the cloud provider, they confirmed that Googlebot traffic passes through an additional crawler-verification step before page content is crawled. When /robots.txt returned 5xx instead of a valid 404 or 200, that verification failed and Googlebot's page requests received a synthetic 5xx generated at the edge — even though origin page content was fine.

The same sequence in words:

1. /robots.txt starts returning 500. The fault sits deep in the 404 and i18n path, so nobody notices — ordinary pages still return 200.
2. Phase one: cached state still valid. Googlebot keeps crawling normally, because it has not yet been forced to re-evaluate the state of robots.txt.
3. Phase two: re-evaluation, and amplification. Once the cached state expires, the edge begins returning synthetic 500 for Googlebot's page requests.
4. Googlebot reduces and then stops crawling. From its perspective, the site has a site-wide availability problem.
5. Fix applied to both the symptom and the cause.

That two-phase delay is precisely why correlating against the most recent deploy misled the investigation: the regression and the visible outage were separated by the lifetime of a cache entry.

What makes this case awkward is that the application bug was real, but its most damaging symptom appeared at a different layer and for exactly one crawler. Stopping the investigation at any single layer would have produced an incomplete conclusion.

Fix

• Add a static public/robots.txt, so the file cannot depend on routing or i18n logic at all.
• Fix the root not-found so it does not depend on the i18n runtime path.
• Constrain the unknown segment so unfamiliar dot-paths return a correct 404.

The two halves do different jobs: the static file is mitigation that stops the bleeding immediately, the routing fix is the actual repair.

Result

• Recovery. The application returned correct semantics for unknown dot-paths and production had a safe static robots.txt. Googlebot fetched successfully almost immediately after the deploy, and page crawling recovered within minutes.
• Vendor confirmation. The cloud provider confirmed the error-amplification behaviour at the hosting edge and acknowledged that it is misleading for crawler troubleshooting.
• Value to the project. It ended the back-and-forth between SEO, frontend and infrastructure by replacing opinion with an evidence-backed failure chain from end to end.
• Value beyond the project. The lesson fed into guardrails for other sites on the same architecture, specifically for SEO-serving routes such as /robots.txt and /sitemap.xml.

Prevention

The incident exposed a monitoring gap: nothing watched the routes that exist specifically to serve crawlers, so a fault there could persist for a long time unnoticed. Three layers, each covering the others' blind spot:

Two design decisions are worth stating explicitly. First, the checks must go through the public domain, not the origin URL — testing the origin would skip the exact layer that caused the incident. Second, they validate body content, not just status code: a robots.txt returning 200 with Disallow: / in it is more dangerous than one returning 500.

Lessons learned

Absence of data is data — but only when it has something to be compared against. What broke this case open was not a stack trace. It was one class of traffic disappearing from the origin while every other flow continued normally.

One well-designed test beats ten arbitrary ones. The two-dimensional matrix located the layer responsible for a site-wide symptom in a handful of commands.

A small fault becomes a large one when it sits on a crawler-serving route. A missing robots.txt is harmless. A /robots.txt returning 5xx is not.

Be willing to correct your own initial conclusion. My first hypothesis leaned heavily towards infrastructure. Investigating properly showed the application regression was an essential link in the chain, and the final report had to reflect both halves honestly rather than the half I expected to find.

A good incident ends with a system, not a patch. The durable value is in the checklists, guardrails and observability added afterwards.

Skills demonstrated

Production incident investigation · Next.js routing and i18n debugging · GCP — Cloud Run, Cloud Logging, Monitoring · hosting edge and CDN troubleshooting · SEO infrastructure and crawler behaviour · observability and alert design · vendor escalation and technical communication