"Perfect" Job – how can I ever leave?

I am currently considering starting my own consulting business so that I can be location independent. That would mean leaving a high-paid, low-stress job that I trained in school for 4 years to attain, in addition to 7 years of working other jobs working up to it. By all appearances, it is a “dream” job – it does something meaningful, I am passionate about the cause, and it provides a nice living for me and my family, with money to spare to donate to causes important to us.

On top of that, I generally only have to work 2-3 hours a day to stay on top of my tasks, leaving time for leisurely workouts, hobbies, and family time. My boss is very hands off and happy with my work. The only downside is I must live in one of two states to stay with this employer and that I’m not particularly challenged; I also dislike several coworkers I have to interact with about twice a week. Otherwise, on paper, there is no reason to ever leave.

How do I reconcile the security and reasonableness of my current job with my desire to be more self-directed in my own business?
I feel “regret in advance” whenever I consider it, and I have the following thoughts:

I’ll never get a job this good again.
It would be a complete shame to walk away from this “easy street” position after spending 11 years to attain it.
When I consider how few hours I work to cover my responsibilities, I’m getting paid $X per hour, which is amazing.
I know I’ll regret leaving this, what’s wrong with me for wanting to?
I am taking this job for granted.

Thank you!