If you are not familiar with Google Alerts, you should check it out. I have been tracking things from Google Alerts for at least 2 years, maybe longer. While I have noticed more things coming in for "DotNetNuke", starting on October 27th I noticed ALOT more alerts for "DotNetNuke" coming in. What did they all have in common? BLOGS. Also a few days ago, while looking into the activity of this blog, I noticed a new user agent I had not seen:
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
If you go to that link, you are redirected to a FAQ page and at the bottom is a section called Feedfetcher. Here is an interesting Q and A:
How do I request that Google not retrieve some or all of my site's feeds?
Since Feedfetcher requests are all user-initiated, it does not follow the typical robots.txt guidelines for robots. For detailed information about how to prevent Feedfetcher from requesting all or part of your site, please see our removal instructions.
Very interesting. I was under the assumption that any "bot", and I will define Feedfetcher as a "bot" regardless of whether it is "user-initiated" or not, should obey robots.txt.
With that said, our feed aggregation module for Venexus Search Engine, called Seamus, does obey robots.txt. I am sure this discussion will come about with the release of VSE, so I decided to go ahead and post it now in preparation. And speaking of Venexus Search Engine...we have made the final compile and are finishing testing tonight...but more on that later.