Hi Plotter,
We live not far from there and the safest and easiest route is as follows
1) leave at 47
2) head for penclawdd (signs for gower)
3) once through penclawdd head for llanrhidian and then go over cefn bryn, little up and down but not as bad as the Bishopston route.
4) once over cefn bryn follow signs for rhosilli and then head for Pitton Cross.
It is really hard to explain accurately and in detail, I will mark it out on a map and scan it for you, this will be more clear especially when trying to navigate, If you wish I will ask the mods to forward your email to me ?
Hope this helps
Ian