I fixed some bugs in the #FediSearch crawler and once again cleared whole index. It will take for about two days to fill it back...
News:
1️⃣ Fixed mastodon api pagination
2️⃣ It now respects #nobot tag in bio
3️⃣ Added robots-parser library for full complience with robots.txt specification (ua: "FediCrawl/1.0")
4️⃣ Now fully respects Mastodon noindex option.
5️⃣ I also added page about opting out fedisearch.skorpil.cz/optout
News:
1️⃣ Fixed mastodon api pagination
2️⃣ It now respects #nobot tag in bio
3️⃣ Added robots-parser library for full complience with robots.txt specification (ua: "FediCrawl/1.0")
4️⃣ Now fully respects Mastodon noindex option.
5️⃣ I also added page about opting out fedisearch.skorpil.cz/optout
This entry was edited (2 years ago)
Archos reshared this.
NoLog.cz 🏴
in reply to Štěpán Škorpil • • •Michal 🇨🇿
in reply to NoLog.cz 🏴 • • •Štěpán Škorpil
in reply to Michal 🇨🇿 • • •The only thing I noticed is some kind of rate limiting on newer peertube instaces.
And large instaces often timeout because of their work load.
Štěpán Škorpil
in reply to Štěpán Škorpil • • •Michal 🇨🇿
in reply to Štěpán Škorpil • • •Michal 🇨🇿
in reply to Štěpán Škorpil • • •{"error":"Search queries pagination is not supported without authentication"}
Štěpán Škorpil
in reply to Michal 🇨🇿 • • •Michal 🇨🇿
in reply to Štěpán Škorpil • • •Every instances with fresh #Mastodon version. It was merged month ago into main branch.
github.com/mastodon/mastodon/p…
Change unauthenticated search to not support pagination in REST API by Gargron · Pull Request #19326 · mastodon/mastodon
GitHubŠtěpán Škorpil
in reply to Michal 🇨🇿 • • •