I recently learned that voting on lemmy is not anonymous. Anyone can get information about who has upvoted and downvoted a post or comment.
In combination with your IP, this is a massive privacy (maybe even physical security) risk. Also, people can target you for your votes.
Sadly, this is something where I would prefer Reddit over Lemmy. Big tech scrapes data from both places anyways, at least Reddit is safe.
Why are you worried about admin abuse? If you are worried that your admin will abuse you, you should switch to an instance you trust more.
How to decide who to trust?
Reputation, word of mouth, history, etc. Same way you decide anything else you consume.
How do you pick where you go shopping? You pick the closest one. Then if it turns out to be bad, you go elsewhere.