Hmm I tried changing the tilt to zero (it was already roughly zero) and significantly changing the index of refraction on the middle window and it had no effect. I am quite sure it is a problem of settings, but I don't know how (or if it is possible) to get the backdrop adjustment settings from the camera settings to apply in the ray trace.
To be more clear, the spherical panorama, when imported, by default, looks tilted due to the curvature of the image, until I make manual adjustments in the "spherical panoramic backdrop" dialogue box (see attached picture). That is what I meant by tilt. For example, by adjusting the "vertical max" slider, I managed to fix the problem, but when I do the ray trace, it is like it ignores the setting and reverts to default. So the manual adjustments to the backdroom only seem to work in the Full Camera but not in the ray trace.