Best way to respond to the family why I don’t want to visit a relative

I prefer not to visit a brother in-laws house. I don’t like comments he makes ie. everyone is stupid and he’s not. He’s made comments, that I interpreted as rude about my choice in home remodeling and my choice for my children’s school. My children frequently wish to visit their family – he has 3 kids. I’m asked to join and I would rather do other things (pilates, read, work, clean house). I am encouraging the families to visit. I just wish not to go every time. What is the best way to respond to my family why I prefer not to go to the brother in laws.
Thank you.