Disconnect after Sex

I have a roommate. Someone that has already told me does not want a relationship with me at this time. I agree with those terms but once in a while we engage in sexual relationships.

I am tolerating the idea of that because I feel sex is also good for me. Although it goes against my values of sex before marriage. With this also comes a feeling bad about myself. Frustration about wanting more but not having it. How can I come to a resolution within myself about this? Where do I start? Not sure yet if my reasons for having it are greater than my reasoning for not having sex with a man I am not married to.

I know I  am affected by being sexually physically with him and feeling the immediate disconnect by him right after. It does not bring us closer together. It does not define or clarify the status of the relationship. It is not a healthy way to connect to this man.
My emotion that drives this is love for myself. If I love myself enough I will stop putting myself on the same path of destruction of self? How is that? Feeling pain because a man will not recognize me after me giving and offering my effort. Not sure what to think and what I want I am sure other people have been in my shoes in dating relationships. How does one come to understand this better?