Brain Drama over my dating life

I am single and dating. I want someone who wants partnership, marriage, and kids, and also someone who shares my values and lifestyle.
A man recently expressed interest and my thoughts are that he’s really great. It feels good to be around him. But I know he doesn’t share a very important lifestyle preference that I have. He’s polyamorous. I’m monogamous.
I’m very attracted to him and want to be able to enjoy a connection with him, but I’m worried that by getting involved with him I will be saying “yes” to something that doesn’t align with me, and therefore potentially hurt down the road when we do part ways.
(I know being hurt is a choice and based on my own thoughts about what it means for a relationship to not work out, but I’ve had enough of these experiences to not want to keep doing this over and over again.)

Why am I setting myself up for something that I know won’t work?
How do I deal with the attraction and desire to pursue something with him even though I know this is not a relationship that will work out?
Is there a way I can have it work out and feel good about it instead of the current anxiety I feel?
Do I just need to say “no” and trust my choice?