When to stop refactoring

When the next change feels easy. Not when the code is beautiful; that's a moving target and the goal is shipping.

I keep a 'refactor budget' for any feature: if I blow through it without making the next step obviously easier, I revert and try a smaller cut.