I think I’ve found the solution, or rather someone else had a similar problem and I’ve tried the same solution.
I disabled Crystal Imaging as suggested by forlaunchcast in the thread Various games crashing after about 30-60 minutes.
I’ve played about 2 hours so far and it hasn’t repeated this HMD glitch on me once (got the game crashing on me fairly often at the moment but I think that is just Fallout 3)