Majora's Mask Time Limit and Running out of Time?

I am thinking of playing The Legend of Zelda: Majora's Mask for the first time. Various reviews mention a time limit. Supposedly, you can slow down time and return to day one. My question is, if you run out of time, and do this, does that mean you have to start the whole game from scratch? If I have to play the whole game again because I'm not fast enough, then I'll pass.