More attacks by dump AI-bots

[Makarius]

On 17/11/2025 11:28, Achim D. Brucker wrote:
>>>    https://isabelle.sketis.net/components
>>>    https://isabelle.sketis.net/repos/isabelle
>>>    https://isabelle.sketis.net/repos/isabelle-release
>> The problem are our hgweb servers (via Apache and its builtin Python 
>> engine), this also applies to https://isabelle.in.tum.de/repos which is 
>> presently unusable.
>> We need to find a proper solution for hgweb + Apache specifically.
> 
> I would not call it a "proper" solution, but I am currently using Anubis 
> (https://anubis.techaro.lol/) with quite some success. Of course, it's an arms 
> race - computing the challenges set out by it are not that expensive. Hence, 
> when enough website use it, the crawlers will implement the challenge solving 
> part ...

Anubis emerged early 2025 as a counter-attack, and I don't like it. An "arms 
race" is war against war, and ultimately won't work.

There must be a proper solution. For us it means that our own programs (or 
"daemons") can access the repository servers, without too much additional 
complication.

I am presently thinking of SSH and maybe RSYNC, as well-known non-HTTP 
protocols. There is also an rsync server that hardly anybody remembers now (we 
actually have one to mirror the Isabelle website).


	Makarius