Is the resolution that I specify in the game settings the resolution for one eye or for both? In other words: would the native resolution of the Rift be achieved at 1280×800 (16:10) or 640×800 (4:5)?
Downsamplig improves the visual quality (but also makes the image slightly blurrier).
You might also want to try 4:3 or 5:4 resolutions which make things easier for vorpX’s Aspect Ratio Correction. There is a possibility though that those aren’t shown correctly on your Rift. If the image looks odd with 4:3/5:4 resolutions, stick to 16:10.
So back to my original question, is the image rendered twice in the specified resolution (in Geometry mode)? And if yes, wouldn’t 4:5 (not 5:4) be the best aspect ratio for stereoscopic side-by-side images in the Rift?
The safest resolution is the native Rift resolution (1280×800).
Besides that, you can try every other resolution you want, and see how that works out. The ideal resolution for vorpX would be a resolution with 1:1 aspect ratio. If you want to experiment with custom resolutions and GPU scaling, you can try to get that working somehow.
If you destroy your Rift or graphics card while doing so, you’re on your own though.