I use both Google and Bing search results RSS feeds on my website and I'm not getting a Feed Informer error message that both sites have blocked all access by Feed Informer. Is something being done to get this unblocked? Please explain. Thanks.
Forums
Feed Informer Support
Can FI get unblocked by Google and Bing?
4 posts-
Posted 10 years ago #
-
We haven't announced it yet, but since recently Feed Informer crawler has started to act with respect to robots.txt protocol (http://www.robotstxt.org/). The robots.txt file for news.google.com (http://news.google.com/robots.txt) discourages any robots (not just us) from visiting /news directory.
As a result of violation of this protocol FI robots will most probably get banned by IP address, sooner or later.Posted 10 years ago # -
So how do we fix this? I need those feeds.
Also, can you check to see if the pantagraph.com site has removed its block? They're trying to do so at my request.
Posted 10 years ago # -
Regarding the robots.txt at pantagraph.com, there is a small correction needs to be made: instead of "User-agent: Feed Informer" it should be "User-agent: FeedInformer" (without space).
Unfortunately, there is no fix for Google News or similar issues, at least for now. Basically, they request us not to do what we do, and we choose to follow their rules, there is no other way around it.
Posted 10 years ago #
Reply
You must log in to post.