Googlebot

Googlebot – Technical SEO

Google's web crawler. It discovers URLs, downloads pages and passes them on for processing. A separate Google service renders the JavaScript.

On this page

Googlebot is Google's crawler. It discovers URLs, downloads the HTML and other allowed resources, and hands the content to the systems that process it for search.

What is Googlebot?

Googlebot follows links, reads sitemaps and revisits known URLs. Google runs several crawlers and user agents depending on the product and purpose.

For regular search, Google mostly uses the mobile crawler. The Chromium-based Web Rendering Service renders the JavaScript afterwards, so crawling and rendering happen as separate steps.

How Google sees your site

Google first processes the downloaded HTML and can then queue the page for rendering. Scripts, network requests or blocked resources can change the final result.

A bot usually won't get content that depends on a click, a scroll or a login the way a user does. Deliver the important indexable information in a way that doesn't rely on those.

How to check the result

  • URL Inspection in Google Search Console shows the index status and the last crawl.
  • The live test lets you compare the current response with what's in the index.
  • Server logs show the actual crawler requests.
  • Comparing the source HTML with the rendered HTML reveals the differences.

A user agent can be spoofed. If you need to confirm it's really Googlebot, use Google's method based on a reverse DNS lookup of the IP address followed by a forward lookup.

See also:Technical SEOJavaScript SEO

Tell me what you need to solve

Tell me what you are working on and what you want to achieve. I propose the next step and a scope of work. We agree on the price before starting.

Fill in the short form, send an email, or message me on WhatsApp if it's urgent.

If I'm not the right person for your project, I'll tell you.