posted Mar 06, 2007

Reading over my web server logs, it's a little annoying how certain big-name web search crawlers will continue to ask for a file years after it's been removed. It's also annoying that some crawlers will get a permanent redirect, but years later, still have to retrieve the original page and then follow the redirect.

Makes it impossible to find any "real" 404s in my error log anymore.


Site Links


All Posts