I fixed some bugs in the #FediSearch crawler and once again cleared whole index. It will take for about two days to fill it back...
News:
1️⃣ Fixed mastodon api pagination
2️⃣ It now respects #nobot tag in bio
3️⃣ Added robots-parser library for full complience with robots.txt specification (ua: "FediCrawl/1.0")
4️⃣ Now fully respects Mastodon noindex option.
5️⃣ I also added page about opting out fedisearch.skorpil.cz/optout
News:
1️⃣ Fixed mastodon api pagination
2️⃣ It now respects #nobot tag in bio
3️⃣ Added robots-parser library for full complience with robots.txt specification (ua: "FediCrawl/1.0")
4️⃣ Now fully respects Mastodon noindex option.
5️⃣ I also added page about opting out fedisearch.skorpil.cz/optout
This entry was edited (1 year ago)
Archos :distros_arch: :matrix: reshared this.
NoLog.cz 🏴
in reply to Štěpán Škorpil :skorpil_cz: • • •Michal 🇨🇿
in reply to NoLog.cz 🏴 • • •Štěpán Škorpil :skorpil_cz:
in reply to Michal 🇨🇿 • • •The only thing I noticed is some kind of rate limiting on newer peertube instaces.
And large instaces often timeout because of their work load.
Štěpán Škorpil :skorpil_cz:
in reply to Štěpán Škorpil :skorpil_cz: • • •Michal 🇨🇿
in reply to Štěpán Škorpil :skorpil_cz: • • •Michal 🇨🇿
in reply to Štěpán Škorpil :skorpil_cz: • • •{"error":"Search queries pagination is not supported without authentication"}
Štěpán Škorpil :skorpil_cz:
in reply to Michal 🇨🇿 • • •Michal 🇨🇿
in reply to Štěpán Škorpil :skorpil_cz: • • •Every instances with fresh #Mastodon version. It was merged month ago into main branch.
github.com/mastodon/mastodon/p…
Change unauthenticated search to not support pagination in REST API by Gargron · Pull Request #19326 · mastodon/mastodon
GitHubŠtěpán Škorpil :skorpil_cz:
in reply to Michal 🇨🇿 • • •