When set to battery, the frdige should have a telltale light on it. On mine, with the EC400 control panel, if the car is idling the fridge light flashes, as soon as revs are applied, the light becomes solid.
When you say the Leisure battery light is amber, how do you know this when you are towing. As others have said, at idle you may not be supplying enough juice.
I did have a similar issue to this, but it affected my fridge rather than leisure battery. In the end, the towbar fitter bypassed the dedicated wiring kit and ran a supply direct to the corresponding cable in the 13 pin setup. In my case, Canbus was restricting the power.