1️⃣ I finally added a method to the crawler, that removes users from index if they disappear. For example when their accounts on instances are deleted or marked as not public. ⚰️ 🗑️ ♻️

2️⃣ I fixed the situation when crawler is redirected to a large file or to an endless stream when searching for nodeinfo api.
This situation used to block the crawler until the memmory run out. Then it crashed and process was restarted. So it also slowed down the crawling.
🐛 ♾️📈

Current statistics of #fediverse reported by #fedisearch:
#Mastodon 222903 public users on 3551 instances
#Pleroma 634 users on 1315 instances (only current beta has needed api)
#Peertube 277613 channels on 1195 instances
#Misskey 23931 users on 402 instances
📈