Fat adaptation

I am losing weight at a pretty rapid pace and would like to include one joy meal each week that includes sugar and/or flour. Will that make my body go out of fat adaptation mode? Will I be starting over to get fat-adapted if I have these foods once a week?