I discovered this a couple months ago while using SEM Rush’s SEO Audit tool for a Real Estate client after her new site launched. The audit came back with about 200 errors. Since the site had just been redesigned by a WordPress Expert (an associate of mine that I referred) we both had a bit of a panicked reaction. So I looked closer at the errors. I tried to confirm them and could only manually confirm a half dozen of the reported errors. This looks like an SEO Dilemma for SEM Rush. I’m rooting for them to fix it.
I called SEM Rush for help. I was told that I need to white list SEM Rush’s Bot IP’s in any place that had security layers on the site, such as CDN’s, Firewalls and Word Fence. Uh Oh! I use Word Fence on all the WordPress sites I work on. If it is not there I’ll even add the plugin. But I was also getting problems from IDX, an MLS service my Real Estate clients utilize. Help desk said, “white list our IP there too.” I knew I was not going to jump through those hoops. I am just not willing to take those extra steps when I can get my Audit data elsewhere and even old school (which pays a lot better). And it is a well established fact that white listing work can lead too SEO nose bleeds.
I wanted to know why I was being asked to white list their Bot IP’s? Where they being blacklisted by Security technologies? On closer look at the errors I noticed that they were actually not “404 errors (page not found)” but they were, the vast majority, “403 errors (access denied)!”
I sent an email to SEM Rush asking what they plan on doing about this problem. What I’m pointing out here is that their bots are so aggressive that they are triggering security on the Web and getting blacklisted by very important sites and technologies. The reply I got was not helpful or even satisfactory.
Hi. We would like to know what semrush is going to do about the ever increasing problem of your bots being blacklisted? These blacklistings are skewing the audit data with 403’s that are being reported as “errors.” I called in about this and your help desk wanted to give me your IPS to whitelist but I have one client that has 3 aspects blacklisting semrush bot. IDX, their host and their wordpress security. As ********* and our clients are all on WordPress, this is a real issue that needs to be resolved some way that doesn’t require us to do white-listing work, which is technical and difficult. How are you addressing this issue/problem? Thank you, Robert
And the reply I got was:
Thank you for contacting SEMrush! Thank you for providing this for us. I have provided your case to the developers as feedback. I do understand the frustrations when seeing the 403 responses and then trying to manually go into hosting providers, CDNs, Plugins, and Firewalls. Our developers consider every recommendation when it comes to the tool. I do apologize if this was not the answer you were looking for. If you need anything else, please email me back. I look forward to your reply!
Obviously skirting the issue. It was not the answer I was looking for, so, after looking for forums or discussions about this issue and not finding any I’ve decided to blog this and see if they’ll get to answering it more proficiently.
You see, the truth is that I am a huge fan of SEM Rush, I have been since their inception, when was that, back in 2008. Always did a great job with providing useful data. But this problem could kill the company, even with all their new and cool betas. If they can’t fix this problem they should just get rid of the features that lead to 403’s reported as “errors.” I am currently shopping for a replacement technology. Several of the ones I like use SEM Rush API for data but not for Audit.
Dear SEM Rush, what are you going to do about this problem? It seems to me that you should be contacting these companies and asking them to white list your IP’s rather than telling Webmasters to do it. Many Web designers and in-house IT folks don’t even know how to do that white listing work.
I hope you get it figured out.
I’ll miss you if you can’t.