Fear of ending things with guys I’m dating

When I’m dating, I usually only see guys once or twice before I end it. I am the Chandler Bing of finding fault over stupid stuff, and I put a LOT of value on looks. It’s possible someone could grow on me over time, but instead I end things immediately because I so dread ending things with someone who’s invested – I even feel sick to my stomach sending the “no thanks” text after a first date. I have no idea WHY I dread it so much. I’ve been out with over a hundred men in the last few years and I’ve only had a small handful lash out when I tell them it’s not going to work. I guess my model is:
C: I am not physically attracted to an otherwise great guy
T: If I don’t end things now, he’ll get invested and either be upset or angry when I end things
F: guilt/fear
A: I end anything I’m not 100% sold on immediately
R: don’t find someone I like

I’ve tried telling myself “it’s none of my business how he reacts” but that doesn’t change the feeling. I guess because I don’t believe it.