.Google's John Mueller addressed a concern concerning why Google.com marks web pages that are refused from crawling through robots.txt and also why the it is actually secure to ignore the relevant Browse Console files regarding those crawls.Bot Web Traffic To Concern Specification URLs.The person asking the question recorded that robots were actually creating hyperlinks to non-existent inquiry criterion URLs (? q= xyz) to webpages along with noindex meta tags that are actually likewise blocked out in robots.txt. What caused the question is that Google is actually crawling the web links to those webpages, acquiring blocked through robots.txt (without watching a noindex robots meta tag) after that getting reported in Google Search Console as "Indexed, though shut out through robots.txt.".The person talked to the observing question:." But below's the significant concern: why would Google index webpages when they can't also observe the information? What's the benefit during that?".Google's John Mueller affirmed that if they can not crawl the webpage they can't observe the noindex meta tag. He likewise creates an appealing mention of the internet site: hunt driver, suggesting to dismiss the results considering that the "common" users will not observe those results.He wrote:." Yes, you are actually proper: if we can not crawl the page, we can not observe the noindex. That claimed, if our company can't creep the web pages, then there's certainly not a lot for us to mark. Therefore while you may see some of those pages along with a targeted website:- concern, the average consumer will not observe them, so I wouldn't fuss over it. Noindex is actually additionally great (without robots.txt disallow), it only suggests the URLs will certainly find yourself being crawled (and find yourself in the Search Console record for crawled/not recorded-- neither of these conditions cause issues to the rest of the website). The integral part is that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's solution confirms the limitations being used the Website: search accelerated search driver for diagnostic reasons. Among those reasons is given that it is actually certainly not attached to the routine hunt mark, it is actually a separate point altogether.Google's John Mueller commented on the website search operator in 2021:." The short solution is that a website: concern is actually certainly not suggested to become comprehensive, neither used for diagnostics purposes.A web site query is a details sort of search that limits the end results to a specific website. It's essentially merely the word internet site, a colon, and after that the web site's domain name.This concern limits the end results to a specific site. It is actually certainly not indicated to become an extensive selection of all the pages coming from that site.".2. Noindex tag without utilizing a robots.txt is alright for these type of circumstances where a bot is actually connecting to non-existent web pages that are getting uncovered through Googlebot.3. Links with the noindex tag will produce a "crawled/not listed" item in Browse Console and also those will not possess a bad result on the remainder of the site.Check out the question and respond to on LinkedIn:.Why will Google mark pages when they can't even view the information?Featured Photo through Shutterstock/Krakenimages. com.