Giving up sugar and flour for life

I’ve been on this journey (no sugar no flour) for 3 years. I’ve tried once a week joy eats. I’ve done once every 2 weeks joy eats. I’ve done intermittent fasting and reached my magic goal weight of 55 kg. Only to binge my way all the way back to 68 kg.
I think the mistake I made when I was at goal weight was to let little bits of unplanned sugar and flour back in and then deal with constant overdesire. That and the ‘anticlimax’ of goal weight let it creep back on and the shame of it coming back brought more buffering.
And then I gave up drinking and food became my primary entertainment on the weekends.
Fast forward to now I am convinced that my life is better when I just don’t eat it. It’s been almost 4 weeks of almost perfect nsnf and I feel so much better. I still have the urges to buffer and I still want to overeat other foods but the urge to binge and the constant craving is gone.
I honestly think I don’t want to eat it again. But I also have a fear that it’s not realistic to never eat it again. I want to be a little scared of it because it helps me to not eat it when it’s around. But I also don’t want to binge for days if i slip up. I don’t want to plan a joy eat because I don’t want to have to deal with all those urges again. Can you help me work on my thinking around this. In the past when I planned a joy eat it took a lot of presence and effort to stop at what I had just planned and then I can see in my food journals the increased urges and often off protocol eating that followed it.
My main focus for the year is to just find peace with food and my body and I think joy eats disrupt that.
I’m thinking the solution to this question is to not plan any joy eats but know that they are always an option if there is something I really want and am prepared for the discomfort that follows.
Your comments and coaching would be very helpful, thanks in advance.