I fixed some bugs in the #FediSearch crawler and once again cleared whole index. It will take for about two days to fill it back...

News:
1️⃣ Fixed mastodon api pagination
2️⃣ It now respects #nobot tag in bio
3️⃣ Added robots-parser library for full complience with robots.txt specification (ua: "FediCrawl/1.0")
4️⃣ Now fully respects Mastodon noindex option.
5️⃣ I also added page about opting out fedisearch.skorpil.cz/optout