Not eating at night and people making comments about it

I already see my own judgment in the title!  For more than 3 years my partner and I aren’t eating at night for various reasons.  We make some exceptions here and there but we just feel better when we don’t.  My father, friends, and people, in general are telling us that “You are exaggerating, you can be flexible a bit! That’s a shame!” and I totally understand where it’s coming from and I’m aware of the manual that I have about them (they should understand and let us do whatever we want/feel like) but at the same time, I find it annoying!  Also, when my dad says for example “I won’t be there much longer and it could be nice if you could make an effort.”  Obviously, he should join Scholars!
Actually, if I think about it, I just want to be able to own my “no”.  I’m not sure how to coach myself around this!  Thank you so much for your precious help!