It happens to me too. I'm not sure how often or if there are any patterns since I haven't been watching too many clips since I got my Env Touch. I have had the same issue with older phones i've had as well too.
I have noticed this whenever I am in an area that I lose EVDO signal while the video is playing like a doctors office. I'll have 1-2 bars of EVDO when i access the video but if I loose it, it has to keep finding the signal to rebuffer.