the 4 strokes are just so brutal right off idle that they wear me out after a few hours.
This is true. When geared like the XC-W, my 530 EXC is a handful on tight/technical stuff. It does wear you out. I try riding in a higher gear and slipping the clutch to smooth it out a bit.
Even the 500 is a bit heavy compared to the two strokes. I prefer quad-trails, double track, dirt roads (more open stuff), so the 530 works great there.