---
title: "Googlebot 500 on Firebase Hosting: Next.js Production Incident Case Study"
description: "Users and every other crawler got 200 while Googlebot got 500 site-wide. The cause: a 404-path regression amplified by crawler verification at the hosting edge."
canonical: "https://tmtuan.dev/en/articles/googlebot-500-firebase-hosting-nextjs"
language: "en"
image: "https://tmtuan.dev/_img/articles/googlebot-500-firebase-hosting-nextjs/googlebot-500-firebase-hosting-nextjs-cover.adc2a628-og.jpg"
updated: "2026-09-18"
tokens: 3356
---

# Case study — How I Traced a Googlebot-only HTTP 500 Across Next.js, Cloud Run and the Hosting Edge

> This case study intentionally omits proprietary source code, internal repository information, cloud project identifiers, operational timestamps, verbatim vendor correspondence and confidential client data. What remains is the investigation method, the failure chain at a technical level, and the operational lessons.

## Context

A production website for a Japanese enterprise client: Next.js on the App Router, running on Cloud Run, with public traffic served through Firebase Hosting.

One property of that setup shaped the entire investigation. The project team could observe the application and Cloud Run directly. The edge layer serving the public domain was largely outside that visibility — there was no direct access to inspect its full internal behaviour.

_Diagram: Googlebot sends a request to the Firebase Hosting edge, which rewrites it to Cloud Run running Next.js. The edge sits between the crawler and the origin._

The public domain passes through an edge layer the application cannot observe directly. A request that succeeds at the origin therefore says nothing about what Googlebot receives on the public domain.

That gap is where the incident lived.

## Problem

Search visibility dropped sharply over a short period. The website itself was working normally for real users.

That combination rules out the usual explanations immediately. It is not a traffic problem and not a content problem — it is an infrastructure problem specific to how crawlers are served.

## Symptoms

Four properties made this hard to trace, and they are worth stating because each one defeats a standard debugging reflex:

**The failure was invisible to users and to conventional monitoring.** Browsers and other bots received HTTP 200. An ordinary uptime check saw nothing wrong.

**The failure left no trace where everyone looks first.** Googlebot's failing page requests never reached the origin, so Cloud Run logs contained no stack trace and no corresponding 5xx.

**Cause and effect were separated in time.** The application regression had shipped well before the incident surfaced. Correlating against the most recent deploy therefore pointed in the wrong direction.

**It was a multi-layer failure chain, not a fault.** Looking only at the application showed one broken response at `/robots.txt`. Looking only at SEO showed Googlebot had stopped crawling. The value was in connecting the two into a single chain of causation.

## Investigation

The guiding principle throughout: each step had to **eliminate a class of hypotheses**, not merely accumulate evidence for whichever one currently looked most likely.

That matters more than it sounds. The initial hypothesis leaned heavily towards infrastructure, and gathering supporting evidence for it would have produced a confident, incomplete answer — the application regression would have been missed entirely.

## Diagnostic experiments

### 1\. A two-dimensional matrix to locate the failing layer

Rather than auditing configuration one setting at a time, I tested the request path against the user agent:

| Request path | Ordinary UA | Googlebot UA |
| --- | --- | --- |
| Through the public domain (edge) | `200` | **`500`** |
| Directly to the Cloud Run origin | `200` | `200` |

The site-wide 500 that Googlebot saw existed **only** at the intersection of public edge and Googlebot. That eliminated normal application page requests as the direct source — but it did not yet explain what condition triggered the edge behaviour.

### 2\. Comparing several crawlers to find the exact boundary

| Crawler | Contains "Googlebot" in UA | After the incident began |
| --- | --- | --- |
| Googlebot | yes | stopped crawling page content |
| AdsBot-Google | no | still working |
| bingbot | no | still working |
| DuckDuckBot | no | still working |

AdsBot-Google is the informative row: also Google, also crawling, but unaffected. So this was not "bots are blocked" and not "the site is down". The behaviour tracked something about how requests carrying `Googlebot` in the user agent were handled on the public path.

### 3\. Building a timeline from logs instead of guessing

Crawler activity ran steadily up to a point and then stopped absolutely, while other bots continued unchanged. A static snapshot would not have shown that. The sharp edge in the timeline indicated a state threshold had been crossed rather than a gradual degradation.

### 4\. A control site

I ran the same tests against a different website on the same architecture. It showed no 500 behaviour for Googlebot, which eliminated "platform-wide outage" as an explanation.

### 5\. From the edge symptom to the real dependency

With ordinary page requests ruled out, I moved to the dependencies Googlebot checks _before_ crawling — `/robots.txt` first. That is where the chain became visible.

## Root cause

An application regression **and** an amplification mechanism at the hosting edge. Neither half explains the incident alone; only together do they produce the full failure chain.

**In the application.** After the site was upgraded to a multi-language flow, unknown dot-paths such as `/robots.txt` no longer fell through to a plain 404. Part of the i18n and 404 logic used **runtime-only request APIs inside a static render context**, which turned what should have been a 404 into an HTTP 500. This was a genuine code regression, and I fixed it directly.

**At the hosting edge.** After escalating to the cloud provider, they confirmed that Googlebot traffic passes through an additional **crawler-verification step** before page content is crawled. When `/robots.txt` returned 5xx instead of a valid 404 or 200, that verification failed and Googlebot's page requests received a **synthetic 5xx generated at the edge** — even though origin page content was fine.

_Diagram: Failure chain in time order: an application regression makes /robots.txt return 500; the hosting edge verifies robots.txt before crawling; phase one, cached state is still valid so Googlebot crawls normally; the cached state expires; phase two, the edge returns synthetic 500 for Googlebot page requests. The chain then branches: Googlebot sees every page request fail and reduces crawling until the fix restores it, while browsers and other bots keep receiving 200 throughout._

Two properties made this hard to trace. The cache lifetime separates the regression from the visible outage, so correlating against the latest deploy points the wrong way. And the right-hand branch never fails — which is why the site looked healthy to everyone except Googlebot.

The same sequence in words:

1. **`/robots.txt` starts returning 500.** The fault sits deep in the 404 and i18n path, so nobody notices — ordinary pages still return 200.
2. **Phase one: cached state still valid.** Googlebot keeps crawling normally, because it has not yet been forced to re-evaluate the state of `robots.txt`.
3. **Phase two: re-evaluation, and amplification.** Once the cached state expires, the edge begins returning synthetic 500 for Googlebot's page requests.
4. **Googlebot reduces and then stops crawling.** From its perspective, the site has a site-wide availability problem.
5. **Fix applied to both the symptom and the cause.**

That two-phase delay is precisely why correlating against the most recent deploy misled the investigation: the regression and the visible outage were separated by the lifetime of a cache entry.

What makes this case awkward is that the application bug was real, but its most damaging symptom appeared at a different layer and for exactly one crawler. Stopping the investigation at any single layer would have produced an incomplete conclusion.

## Fix

- Add a **static `public/robots.txt`**, so the file cannot depend on routing or i18n logic at all.
- Fix the **root not-found** so it does not depend on the i18n runtime path.
- **Constrain the unknown segment** so unfamiliar dot-paths return a correct 404.

The two halves do different jobs: the static file is mitigation that stops the bleeding immediately, the routing fix is the actual repair.

## Result

- **Recovery.** The application returned correct semantics for unknown dot-paths and production had a safe static `robots.txt`. Googlebot fetched successfully almost immediately after the deploy, and page crawling recovered within minutes.
- **Vendor confirmation.** The cloud provider confirmed the error-amplification behaviour at the hosting edge and acknowledged that it is misleading for crawler troubleshooting.
- **Value to the project.** It ended the back-and-forth between SEO, frontend and infrastructure by replacing opinion with an evidence-backed failure chain from end to end.
- **Value beyond the project.** The lesson fed into guardrails for other sites on the same architecture, specifically for SEO-serving routes such as `/robots.txt` and `/sitemap.xml`.

## Prevention

The incident exposed a monitoring gap: nothing watched the routes that exist specifically to serve crawlers, so a fault there could persist for a long time unnoticed. Three layers, each covering the others' blind spot:

| Layer | Mechanism | Catches | Blind spot |
| --- | --- | --- | --- |
| Prevention | Smoke test in CI/CD after every deploy, checking `/`, `/robots.txt` and `/sitemap.xml` with a Googlebot user agent | Faults from code or deploy, blocked at the pipeline | Does not run when there is no deploy |
| Active | Periodic synthetic monitoring through the public domain with a custom user agent, checking status **and** body | Any cause, including infrastructure changes — and the "no traffic at all" state | Uses a simulated UA, not a real crawler |
| Observational | Log-based alerting on the hosting edge's request logs | What real crawlers are actually receiving at the edge | Only visible while requests still arrive |

Two design decisions are worth stating explicitly. First, the checks must go **through the public domain**, not the origin URL — testing the origin would skip the exact layer that caused the incident. Second, they validate **body content, not just status code**: a `robots.txt` returning 200 with `Disallow: /` in it is more dangerous than one returning 500.

## Lessons learned

**Absence of data is data — but only when it has something to be compared against.** What broke this case open was not a stack trace. It was one class of traffic disappearing from the origin while every other flow continued normally.

**One well-designed test beats ten arbitrary ones.** The two-dimensional matrix located the layer responsible for a site-wide symptom in a handful of commands.

**A small fault becomes a large one when it sits on a crawler-serving route.** A missing `robots.txt` is harmless. A `/robots.txt` returning 5xx is not.

**Be willing to correct your own initial conclusion.** My first hypothesis leaned heavily towards infrastructure. Investigating properly showed the application regression was an essential link in the chain, and the final report had to reflect both halves honestly rather than the half I expected to find.

**A good incident ends with a system, not a patch.** The durable value is in the checklists, guardrails and observability added afterwards.

## Skills demonstrated

Production incident investigation · Next.js routing and i18n debugging · GCP — Cloud Run, Cloud Logging, Monitoring · hosting edge and CDN troubleshooting · SEO infrastructure and crawler behaviour · observability and alert design · vendor escalation and technical communication

## Related services

- [Cloud Infrastructure](https://tmtuan.dev/en/services/gcp-cloud-infrastructure)
- [Production Troubleshooting](https://tmtuan.dev/en/services/production-troubleshooting)

## Related projects

- [Dr.Well-B](https://tmtuan.dev/en/projects/dr-well-b)
- [TaxPal](https://tmtuan.dev/en/projects/taxpal)
- [GO Inc](https://tmtuan.dev/en/projects/go-inc)
- [Wonderia](https://tmtuan.dev/en/projects/wonderia)

## Structured data (JSON-LD)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Article","@id":"https://tmtuan.dev/en/articles/googlebot-500-firebase-hosting-nextjs#article","headline":"How I Traced a Googlebot-only HTTP 500 Across Next.js, Cloud Run and the Hosting Edge","description":"Users and every other crawler got 200 while Googlebot got 500 site-wide. The cause: a 404-path regression amplified by crawler verification at the hosting edge.","datePublished":"2026-09-18","dateModified":"2026-09-18","inLanguage":"en","url":"https://tmtuan.dev/en/articles/googlebot-500-firebase-hosting-nextjs","mainEntityOfPage":"https://tmtuan.dev/en/articles/googlebot-500-firebase-hosting-nextjs","image":"https://tmtuan.dev/_img/articles/googlebot-500-firebase-hosting-nextjs/googlebot-500-firebase-hosting-nextjs-cover.adc2a628-og.jpg","author":{"@type":"Person","@id":"https://tmtuan.dev/#person","name":"Tran Manh Tuan","url":"https://tmtuan.dev/en"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://tmtuan.dev/en"},{"@type":"ListItem","position":2,"name":"Articles","item":"https://tmtuan.dev/en/articles"},{"@type":"ListItem","position":3,"name":"How I Traced a Googlebot-only HTTP 500 Across Next.js, Cloud Run and the Hosting Edge","item":"https://tmtuan.dev/en/articles/googlebot-500-firebase-hosting-nextjs"}]}]}
```