Hard time understanding when to be in relation with someone and when decide not seeing them


I am really trying to make the shift from “a few people are difficult to connect with/I’ll never get along with them,” to the concept that circumstances and people are neutral. If you always find a way to think neutral or positively of a person, how come you have sometimes to put limits to people? If I am the only one hurting myself, I could change my thoughts anytime and would never have to leave any situation again? How can I decide when not to stay in a situation or with a person?

Also, I find it exhausting to think if the person does that, I’ll do that, if she does this, I’ll do this, for people I have to interact with regularly at work.

Thank you for your help