Anticipating A Breakup

I’ve had a relationship with a married man for the last 14 months. I was married too, but have just recently gotten divorced. I didn’t leave my husband for the other guy. I fully realize that I’ve started having extramarital relationships as a way of buffering my unhappiness in my marriage. I’ve done a lot of work on that part and came to the conclusion that I could learn to be happy in my marriage but I didn’t want to. So now I’m single but still in my relationship with the married guy. We love each other and there are a lot of great things about our relationship. I know he has some similar thoughts about wanting to leave his marriage but feels trapped because they have young children, etc. My problem right now is that I have the thought that my relationship with him will eventually have to come to an end. Here’s the current model: because he might never actually be available and if I were, he might not actually be someone I want to have a serious relationship/ partnership with.
So here’s the current model:
C – relationship with A
T – eventually this relationship will have to end because A will probably never be available or even if he were to get divorced, he’s probably not someone I would want to have a serious relationship/partnership with
F – sad, anxious
A – dwell on how I’m going to eventually lose him, not enjoy the present time that we do have together
R – basically lose him ahead of time in my mind
I want to get to a place where I just enjoy this for what it is now, but the sadness just continues to come up. In a separate but related model I also think “he will never leave his wife and will just have another side girlfriend after me” which makes me feel jealous. Then I dwell on that and again don’t enjoy what we have now/don’t value myself for being amazing and desirable.