You can choose to federate with a specific server. I believe some mastodon servers would honour requests to only share with specific accounts, but that’s it.

You could possibly have some encryption key shenanigans go on at the client side and build it ontop of the fediverse. It might be possible.