Googlebot is Google's crawler. It discovers URLs, downloads the HTML and other allowed resources, and hands the content to the systems that process it for search.
What is Googlebot?
Googlebot follows links, reads sitemaps and revisits known URLs. Google runs several crawlers and user agents depending on the product and purpose.
For regular search, Google mostly uses the mobile crawler. The Chromium-based Web Rendering Service renders the JavaScript afterwards, so crawling and rendering happen as separate steps.
How Google sees your site
Google first processes the downloaded HTML and can then queue the page for rendering. Scripts, network requests or blocked resources can change the final result.
A bot usually won't get content that depends on a click, a scroll or a login the way a user does. Deliver the important indexable information in a way that doesn't rely on those.
How to check the result
- URL Inspection in Google Search Console shows the index status and the last crawl.
- The live test lets you compare the current response with what's in the index.
- Server logs show the actual crawler requests.
- Comparing the source HTML with the rendered HTML reveals the differences.
A user agent can be spoofed. If you need to confirm it's really Googlebot, use Google's method based on a reverse DNS lookup of the IP address followed by a forward lookup.