Polygyny- man married to woman open to 2nd wife

I’m the long term friend of this man who presented me with an option to be a second wife.  He says I’ve damaged our relationship once again because I pulled away for a period in hopes of landing a separate relationship for myself.  When I do this distancing he tells me it’s hurtful.  My thoughts are:  “He’s married with children.  He’s perfectly fine.”  Am I wrong for thinking this?  Heck I’m the one who is alone.