Not being a College Graduate as a deterrent I can’t seem to clean up

I know this has been touched up on on several instances but this thought of not being a college graduate keeps coming up as a negative in my mind and I need help resolving it please.
Be it in informal or official formats, I perceive people change when I say I did not finish college, it seems the dialogue deteriorates or even shifts. How do I work through this, please? I know SO many people make it without one, why am I so stuck there?