The $600 Story: How Googlebot Set an AI Blog's Token Bill on Fire
By Arash Latifi — Founder, The DevArt
A generative demo built a fresh page on every request; Googlebot kept finding new routes forever. The result: $20 of Gemini tokens a day and dozens of fake pages in Google's index. Lessons for anyone with AI-powered pages.
TL;DR — A hobby demo that generated a brand-new page with Gemini on every visit sent Googlebot into an infinite discovery loop: endless crawling, endless token burn, and eventually AI-generated fake content sitting in Google search results. Total damage: $600 in a month.
We usually worry about Google not coming to our site. A post by developer Steven Yue this month showed the opposite failure mode: Google coming too much — on a site where every visit costs real money.
What happened
Steven had built two "Generative UI" demos, including a Hacker News clone where Gemini generated the content on the fly — posts, users, comments, all made up per request. Fun project, until his Google AI Studio bill arrived: $600 in a month, roughly $20 of tokens per day.
The culprit, per the logs: Googlebot
Heroku logs showed nearly all traffic to the HN clone coming from a single IP. A reverse lookup resolved it to crawl-192-178-7-168.googlebot.com — Google's own crawler.
Why did the crawler go haywire? Because the site rendered different content on every request:
- Navigation links changed each time, so Googlebot believed it kept discovering new pages and followed them
- Every new page meant a fresh Gemini API call — fresh tokens, fresh cost
- The loop never ended: crawler discovers → site generates → new routes appear → crawler discovers again
The second disaster: fake content in the index
Then came the sour part. Curious what Google made of all these fabricated pages, Steven searched — and found Gemini's made-up news posts sitting in Google's search results, ten-plus pages of them. A generative site with zero oversight had polluted the index with synthetic content.
Lessons for real sites
You probably don't run a generative HN clone. But if any part of your site — translation, product descriptions, a chatbot, summaries — sits on a paid LLM, the same scenario can repeat on a smaller scale:
| Risk | Symptom | Fix |
|---|---|---|
| A crawler that creates cost | Server logs full of Googlebot hits on expensive routes | Block heavy routes in robots.txt; cache generated pages |
| Phantom pages in the index | Search results show pages from your domain you never built | Audit indexed URLs; noindex or remove |
| Bot traffic that looks like traffic | Impressions up, conversions flat | Compare logs vs analytics; segment bot traffic |
The financial lesson: more visits ≠ more revenue. This demo had steady hourly traffic — and every visit charged its owner. If any route triggers a paid API on render, put limits in place: server-side caching, rate limiting, or a cheaper version served to crawlers.
Checklist if you serve AI-generated content
- [ ] Check server logs: which crawlers and IPs hit your expensive routes most?
- [ ] Exclude expensive generative routes in
robots.txtor serve a cached version - [ ] Set a daily spending cap on your model API (AI Studio and most providers support this)
- [ ] Search
site:yourdomain.com— any indexed URLs you never built? - [ ]
noindexgenerative pages unless you actually want them indexed - [ ] Segment bot traffic in analytics so real numbers don't fool you
Why this matters even without generative pages
Two takeaways apply to everyone. First: everything you serve to a crawler has a cost and a consequence. Pages that accidentally get indexed — internal search parameters, test pages, unbounded filter URLs — build the same endless loop at smaller scale. They don't burn tokens, but they eat your crawl budget.
Second, as the author notes, AI bots are now a meaningful share of site traffic — and it often brings zero clicks. Deciding who gets to read what is now a business decision, not just a technical one.
FAQ
Q: How do I know if Googlebot is burning my money?
Check server logs and count crawler requests against expensive routes (API calls, heavy renders). If your site generates paid content per request and the crawler hits it often, the pattern is already happening. Simple tools: grep on logs plus per-route request counts.
Q: What's the right way to keep generated content out of the index?
noindex for pages you don't want indexed, robots.txt for paths crawlers shouldn't touch at all. They differ: robots.txt blocks crawling, but a linked page can still appear in results without content; noindex forbids indexing but the page must remain crawlable for Google to see the tag.
Q: How do I control model API costs?
Set caps. AI Studio and most providers support daily budgets. Add server-side caching for repeated outputs — if a page will be viewed ten times, generate it once and serve the same result.