Approximately once a month (or perhaps every 2 months) my web sites record a request for a page such as:
This is Yahoo checking to see that the web site correctly returns a 404 for pages that don't exist. To confirm that it really is Yahoo you should do a lookup on the IP where the request came from and check that it's INKTOMI CORPORATION.
I assume that this makes for better indexed web sites as the search engines can rely on your site to return the appropriate error codes for pages that have moved or are no longer there.
Here is some trivial information about the Inktomi bot's visit to the previously mentioned site. The first visit was recorded today (6/21/2008) at 6:31am ET and the last at 12:56pm. There were a total of 9 visits. The shortest time between visits was 9 minutes and the longest 79 minutes with an average of 48 minutes. Each IP address that each request came from was unique but they all fell in the 72.30.215.* block.
I've decided to try and table the visits to see exactly how often they check: