On any online space, you should consider who you give power to. Who has the control over who you choose to associate with? Power doesn't corrupt. It reveals.
People in real life do choose to avoid dangerous or toxic places.
All that the instances who sign the fedipact are doing is signalling to some of us that somewhere is safe for folk who don't want to engage with Facebook at all.
This is a Freedom of Association issue, not a Freedom of Speech issue.