I find it amazing that it took me this long to figure this one out, but here it is: if one is developing on Visual Studio, bypassing the F5 and just testing directly on the site will knock between 1 second to or 5 per test. This is especially useful if it is very apparent whether the change has worked or not by testing the functionality.
It doesn't seem like a lot, but those 5 seconds build up after repeated iterations. Also, those 5 seconds can be enough to break your concentration on what are you looking for, meaning that the true time lost could be greater is we take into account the seconds needed to refocus on the task.
By testing it this way, we only incurred the time penalty of starting the debugger when we actually need to step through the code or to debug per se.
So when the change will work or not, avoid the debugger: build and test, build and test.