Google has updated its open source robots.txt parser code on GitHub the other day. Gary Illyes from Google pushed the update yesterday morning to the repository there....
-
Google Robots.txt Parser Updated In GitHub
seroundtable.com