Industry News – Google’s robots.txt parser is now open source
Today, we announced that we’re spearheading the effort to make the REP an internet standard. While this is an important step, it means extra work for developers who parse robots.txt files.
from Pocket https://ift.tt/2JiCpHk
Read the full story here – IFTTT