In an early July blog post on the evolution of the Robots Exclusion Protocol (REP), Google made two big announcements: moving the REP towards “Internet Standard” status and open sourcing the robots.txt parser. Today we’re announcing that after 25... read more
Have any questions or comments about this article?
