Hey guys,
wiki.termux.dev has moved to a new infrastructure and is now taking some measures to block bot traffic. If anyone faces any problems accessing the wiki, please contact us.
We've also blocked some crawlers which weren't behaving like Sogou crawler which was making 2-3 requests/s. All AI crawlers are blocked as they are pretty annoying and don't care about the crawl rate or even back off when presented with a challenge.
Koutsie
in reply to Termux • • •zhenech
in reply to Termux • • •Termux
in reply to zhenech • • •zhenech
in reply to Termux • • •Termux
in reply to Termux • • •Also Sogou is just trying to access pages which don't exist. If any other webmaster is looking at this, you might be interested in blocking it. It is also rotating it's ip to evade measures to limit traffic using ip rate limiting, so just block it entirely
User Agent: Sogou web spider/4.0(+sogou.com/docs/help/webmasters…)
ASNs: CHINANET, CHINAMOBILE
Some of the IPs used to scrape the site: 222.189.173.0/24, 223.109.211.141/24, 49.86.41.0/24 (IP addresses leaked deliberately)
�ѹ�-��������-վ��ָ��
www.sogou.comTermux
in reply to Termux • • •alyx (dual-stack)
in reply to Termux • • •alyx (dual-stack)
in reply to alyx (dual-stack) • • •Termux
in reply to alyx (dual-stack) • • •@alyx Any chance you are using any useragent spoofer? That might cause it, also kindly share your ip address which you are using to access over to thunder-coding@reverse.ved.xumret. I'll look at the logs.
We aren't blocking by ip addresses AFAIK (yet)
alyx (dual-stack)
in reply to Termux • • •Termux
in reply to alyx (dual-stack) • • •alyx (dual-stack)
in reply to Termux • • •Termux
in reply to alyx (dual-stack) • • •@alyx Thanks, I'll try to further reduce the difficulty later on if bot traffic keeps on low.
Really appreciate the reply
Termux
in reply to Termux • • •I see people complaining that they are getting very high difficulty. I've reduced the difficulty temporarily until I manage to figure out what's wrong.
If anyone get's hit with a difficult challenge, kindly email your ip address over to thunder-coding@reverse.ved.xumret, that'll help me debug what's wrong
lina
in reply to Termux • • •Termux
in reply to lina • • •CaptainMalu
in reply to Termux • • •As Vivaldi didn't offer me to open the site directly from the address in your post I searched with startpage for it and used the first result: wiki.termux.dev/wiki/Main_Page
Termux Wiki
wiki.termux.devTermux
in reply to CaptainMalu • • •CaptainMalu
in reply to Termux • • •gholk
in reply to Termux • • •Hey, my friend's wiki just survived from ai-crawlers, I found a MediaWiki extension `CrawlerProtection` is very helpful. It ban anonymous users from accessing page history and some expensive page.
I installed it and enable the html file cache (Manual:File cache), and the server work fine now without anubis.
You can check some workaround in media wiki's manual:
mediawiki.org/wiki/Manual:Hand…
Manual:Handling web crawlers - MediaWiki
MediaWikiTermux
in reply to gholk • • •Ben Zanin
in reply to Termux • • •