What to do when other people punish/withdraw as a result of their model

I do believe that other peoples’ actions are the result of their models and thus understand that I don’t have to buy into the “I please you so that you treat me in a certain way” model, but what do you do when the other person tells you that you ARE responsible for their actions e.g. “you did this and it really hurt me and now I’m pissed off” and then withdraws or acts out their anger at you?

It hurts to be treated like that, even though I know it’s coming from their model – and the fact that I don’t like others acting out their bad feelings on me has definitely contributed to me not prioritizing my own needs (something I want to change).

I’m wondering if it would end up meaning that I let that person go, or if there is a boundary I can set that doesn’t feel controlling – particularly with the people who withdraw – “If you withdraw from me I will —“. What is my action? Stop being friends? No longer contact them? Don’t re-engage when they (invariably) come back?

It’s easier for me if there are behaviours I can point to like “If you raise your voice at me I will hang up the phone”, but when people just stop calling/ responding/ reaching out what is there to do? Even though I know it’s a result of their thoughts about me, it still hurts to lose them (even temporarily).