Why do I have an issue with my ex drinking?

My ex has always been a drinker. I’m not. I just don’t like the way alcohol make me feel. My parents didn’t really drink (no childhood issues with alcohol). I have no idea why I have an issue with people who drink to excess. My current partner can drink up to 6 beers (I’ve never seen him have more) and I don’t even notice if he is drunk or not.
My ex goes out and gets drunk when he has our children. They are okay to be left alone and don’t seem to really care. They used care a little about him going out but now I think they are used to him doing it.
But I care. I care that he prioritizes drinking/socializing over spending time with them. He often asks me to have them on his time so he can go to events, get wasted and not have to deal with the children in the morning. I’m fully aware I can say no to him but I enjoy having my children extra time and we are fluid with our custody arrangement and help each other out when we can.
I am fully aware he is allowed to do what he wants and I need to think happy thoughts about his drinking to not let it annoy me.
I’m just curious to get to the bottom of why it does annoy me…. at the end of the day it does not affect me.