Is it too late?

I’ve had a product-based business for about 3 years. Because my product-based business doesn’t bring in enough money to cover my living expenses, I’ve been freelancing part-time for two years.
When I originally began freelancing, I was kind of naive and slightly desperate for the money so I accepted very low pay. Now that I’m more conscious of the value I give to my clients, I would like to ask for a pay increase.
Part of me thinks that if they say “no” then that’s a sign I need to just walk away from client work as a whole and put all of my focus on my own business…but I wouldn’t be able to afford it.
My biggest hesitation is I’ve always been a fast worker so I can complete tasks very quickly and my clients always ask me things like “how long did it take you to finish ___?”, so I have a hard time asking for more money when they know I can do the work quickly.
I’m not sure if it’s too late to change my prices or if I should just settle for what I’m being paid and just wait until my main business makes enough money.