How To View A Website As A Googlebot In Chrome -BlackhatWorld
This blog post will explain how to create your own Googlebot model in Chrome to troubleshoot challenging technical SEO difficulties. You must install a web browser designed specifically for Googlebot browsing. Using a user-oriented browser extension is frequently close enough for SEO audits, but further measures are required to get as close to replicating Googlebot as possible.
Googlebot model in Chrome assists in fast replicating and debugging complex technical SEO issues. This method is known as the Chromebot technique.
What Is The Chromebot Technique? `
The Chromebot technique is a straightforward non-code solution that allows a human to customize the Chrome setting to behave like a Googlebot crawler. It can assist SEO professionals in identifying specific crawling and indexing difficulties on a website.
Why Use Chromebot Technique?
This technique is frequently used at DeepCrawl to troubleshoot numerous customer crawling and indexing issues. It’s a basic but effective non-code strategy for helping technical SEOs think like search engine crawlers rather than a human.
When Googlebot users request pages, many websites can behave awkwardly.
Why Should A Website Be Viewed As Googlebot?
There are several factors that determine how to view a website as Googlebot:
- Previously, internet servers delivered complete websites rendered on HTML to web browsers. But now, many websites are rendered client-side inside the web browser for Chrome, Safari, or any other browser that a search bot uses. This will imply that the person’s browser and the system should perform the effort to produce a webpage.
- When we view a website as Googlebot, we can detect differences between human and search engine observation. What Googlebot sees does not have to be the same as what a person using a browser sees, but the primary navigation and the content you want the web page to rank for must be identical. Therefore, for a proper technical Seach engine optimization audit, we must look at what the most popular search engine sees, just like Google.
Why Use Chrome To View Websites Like A Googlebot?
There are several advantages to using this strategy, which is discussed below:
1. Debugging In Google Chrome
When attempting to find third-party web crawling tools, we say Chrome is the best platform to do all these activities. Chrome assists you in understanding and debugging difficult situations.
Google Chrome can be your favorite non-SEO debugging tool, and when properly configured, it can even replicate Googlebot to test what crawling tools are picking up.
2. Unique SEO Insights
It only takes a few minutes to rapidly identify why they are crawling or indexing issues using Google Chrome to analyze web pages like Googlebot. Anyone can use this method to immediately debug possible crawling and indexing instead of waiting for a web crawler to complete processing. Try to determine the severity of an issue using the crawling data.
3. Googlebot Doesn’t Act Like A Human
The internet is becoming increasingly complex and dynamic. When troubleshooting crawling and indexing issues, remember that you are human and Googlebot is a machine. Many current websites approach these two types of visitors differently.
Google Chrome, which was developed to assist people in web navigation, may now assist a human in seeing a site like a bot.
Why Use A Separate Browser To View Websites As Googlebot?
There are several factors for using other browsers instead of Chrome that are discussed below:
It saves time to have a dedicated browser. Get a sense of how Googlebot sees a website in seconds without relying on or anticipating other tools. While inspecting a website that serves different content between browsers and Googlebot, tries to switch between the normal user-oriented browser and Googlebot more frequently than usual. However, fixed user-agent switching with a Chrome browser plugin was insufficient.
2. Improved Accuracy
Browser extensions can influence how websites look and function. This method keeps the number of extensions available in the Googlebot browser to a minimal level.
If you’ve been blocked from websites for faking Googlebot and had to email them your IP address to get the block removed. It’s simple to forget to change Googlebot’s spoofing off between looking classes, leading to websites not performing as expected.
How To Setup Googlebot Simulator?
Following are the best practices to create your own Googlebot Simulator:
1. Download Google Chrome
Download Chrome Canary instead of your own Google Chrome Browser. If you’ve moved to Firefox, then use Google Chrome. The primary reason is that you will be altering browser settings, which can be inconvenient if you forget to reset them or have multiple tabs open. Save time by just using Canary as your dedicated Googlebot simulator.
2. Use A VPN
If you are not in the United States, ensure you have access to a Virtual Private Network (VPN), so you may change your IP address to one in the United States. This is because Googlebot crawls from the United States by default. To replicate crawl behavior accurately, you must pretend to be viewing a site from the United States.
3. Chrome Settings
Once you’ve downloaded and installed all the configurations, it’s time to customize Chrome settings.
4. Use Web Dev Tools
The Web Developer Tools UI is essential for viewing your website as Googlebot. To ensure you can navigate the console, open the Web Dev Tools in a separate window. Keep in mind that your DevTools windows are linked to the tab in which you opened it. When you close that tab in Google Chrome, it also seals the settings and DevTools windows.
To accomplish this, simply follow these steps:
- Click inspect element (CTRL+SHIFT+I) when right-clicking on a web page.
- Navigate to the right side of the screen, click the three vertical dots, and then select the far left dockside option.
A user-agent string allows apps to identify themselves to servers or networks. To replicate Googlebot, we must update the browser’s user-agent to inform websites that we are Google’s web crawlers.
To unlock the network condition tab in DevTools and update the user-agent, use the Command Menu (CTRL+Shift+P) and input “Show network conditions.”
5. Enable Stateless Crawling
Googlebot crawls web pages without saving their status between page loads.
According to the Google Seach Developer documentation, each new page crawled utilizes a new browser and does not rely on the cache, cookies, or location to discover and crawl websites.
Disable The Cache Command Menu
When DevTools is open, press CTRL+Shift+P and type “Disable Cache” to turn off the cache.
Navigate to chrome:/settings/content/location in your browser using Chrome. Activate “Blocked” by switching “Ask before accessing” to “On.”
6. Disable Service Workers
Googlebot disables interfaces that rely on the Service Worker standard. This means it avoids the Service Worker, which may cache data and gets URLs from the server.
Navigate to the Application Panel in DevTools, then to Service Workers, then check the ‘Bypass the network’ option. When enabled, the browser is always compelled to request a resource from the web rather than a Service Worker.
8. Network Panel
Finally, network panel setup is a critical approach for configuration. As Googlebot, you’ll spend a lot of time in this section of DevTools. The Network panel ensures that resources are being fetched and downloaded. This panel lets you see the metadata, HTTP headers, content, and so on of each URL downloaded when you request a page.
However, before we can check the resources like HTML, CSS, and IMG, they should be uploaded from a server like Googlebot. We must adjust the headers to present the most critical information in the panel.
Navigate to the Network panel in DevTools. Right-click on the column headers in the panel table and pick the headings given below to add as columns in the network panel.
9. US IP Address
Your Googlebot emulator is almost ready once you’ve changed the Network panel headers in Chrome DevTools. If you desire to use it immediately, you must change it to a US IP address.
Googlebot originates in the United States of America. As a result, when utilizing your Googlebot simulator, always change your IP address to the United States. It is the best approach to learning how Googlebot interacts with your website. For example, if a website blocks users with US IP addresses or geo-redirects visitors based on their location, Google crawling and indexing may be affected.
How About Viewing A Website As Bingbot?
To make a Bingbot browser, combine the most recent version of Microsoft Edge with the Bingbot human agent. Bingbot is identical to Googlebot regarding what it does and does not assist.
Yahoo! Seach, DuckDuckGo, Ecosia, and other search engines like Google are either powered by or primarily based on Bing search. Therefore, Bing is responsible for a larger share of searches than many people realize.
To wrap up this blog post, we believe the simplest way to scan webpages like Googlebot is to use an existing browser to simulate Googlebot. It’s also free if you already have a desktop computer that can run Chrome or Canary.
Read Out the Most Interesting Topics Links “Click On links” 👉