We are hosted by III and we’ve experienced a significant uptick in our site being swarmed by web crawlers to the point that Leap (we are a Polaris library) becomes unusable. The reliable indicator is an “unable to connect to ERMS” error message when logging in to Leap. We’ve had three incidents in the last month. Innovative suggested setting up a geofence, but when I asked for the problem IP addresses, only one of the three incidents involved IP addresses from outside the U.S.–the others were Google and OpenAI/GPTBot. Are other hosted libraries experiencing this problem lately? If so, has anyone gotten Polaris to set up anything for their server that has helped?
What is the domain/name URL of the hosted sites? I don’t need the full thing; I’m just wondering if it is in your library’s domain or not. If it is in your domain, you might have some additional options available to you.
Regardless, it seems like we’re hearing this enough that it should probably be something that the @steering-committee should bring up (they probably already have but this gives them more fuel) with Clarivate to address to make sure they are addressing it in a more systematic way.
Hi Emma, We here at GMILCS are having similar issues, though ours seem to happen almost daily. Luckily, things seem to resolve themselves, requiring only a refresh or reboot at the moment.
Stark Library (a Sierra library also hosted by Clarivate) has had at least two severe week+ long bot incidents, the first occurring in early Feb 2025 and another a couple of months ago. At IUG 2026, I learned that Clarivate was implementing Cloudflare on its hosted systems this summer. Does anyone know the status of that rollout?
1 Like
All these replies are interesting, thank you!
Bob, I’m sorry to hear Stark’s incidents have lasted so long. Has III not been able to identify and block the originating IPs?
I’m glad to hear Clarivate is planning to implement Cloudflare. I too hope for a status update on that. I put in a general “what are you doing about bots” ticket and got a very generic reply just today that didn’t tell me much:
“Bot and crawlers are being addressed incrementally to avoid blocking patron or otherwise ‘good’ traffic. If traffic is being blocked wholesale/in bulk, it’s much more likely library or patron IP’s will be blocked in error so we are hyper-aware and sensitive to what we’re blocking basically. We are adding additional entries to the robots file that hopefully should prevent or limit this activity as well.”
Hi Wes,
The domain will be something like libraryname.polarislibrary.com, ex. ours is irving.polarislibrary.com. Sadly not in our domain.
Yeah, then it will rely on Clarivate putting it behind a reverse proxy to protect like Cloudflare or bunny.net.
The fact they think the bots are going to respect robots.txt is WILD to me.
They’ve been playing a game of Whack-a-Mole, because none of the origin IPs are used for very long before the bad actor(s) switch to a different IP. We use BiblioCommons for our discovery layer, website CMS, mobile app and events calendar, and at one point Clarivate got a little overzealous with the IP blocking and blocked requests coming from BiblioCommons, disabling the public catalog in the process.
If these miscreants/ne’er-do-wells/scofflaws have a profit motive, I’d like to understand what that is. Is some incredibly well-heeled and possibly evil media retailer, most likely with a supervillain-style clean shaven head (none come to mind at the moment) attempting to drive consumers away from the free lending model of public libraries and toward retail? Nah, that’s a crazy idea.