That's why you do it slowly. People don't really notice the subtle changes... Even when they happen over time, people who see you constantly will never really acknowledge them. But those who you meet, and those who don't see you after an amount of time, will. But they aren't the ones who'd expect you to act a certain way, in comparison to those who see you constantly.

Then keep to it, if it's going to be rewarding for you... But if it's truly troublesome to you, you should also decide whether it's rewarding enough.