Hello and welcome to the communities,
when the problem occured for the first time I also tried to run an installation with an older version as upgrade, but for me it didn't work. I had to completely reinstall the host.
But for you chances might be good that you could at least go back to build 799733, when you select this at the boot prompt. After that you could only apply the update, that get's you to the "somehow" working build. But if I were in your shoes and you want to really rely on the config, then I'd recommend to install ESXi 5.0.