Driving any vehicle (or outfit) requires the full attention of the driver. Distractions of any sort should be avoided.
Cris and others have pointed out there are many distractions, all of which will diminish the drivers concentration and ability to drive.
The majority of the driving laws were introduces in time when there were less distractions than now, in a time before mobile phones or Sat Navs.
It is true that there were other distractions, such as tuning a radio, or lighting a cigarette, but in terms of time and the mental effort required to perform these tasks they were completed relatively quickly, and the drivers attention and hand can return to driving. To a large extent these became second nature requiring less mental effort.
By comparison, a conversation takes a relatively long time, and demands that the contributors have to listen and assess the information they receive, and then try to formulate a coherent response. This is a demanding process and needs considerable brain power to achieve it. We all know that it is a trait of human nature that if we need to concentrate hard or harder on a particular activity, speech is one of the first activities we drop to free up our mental agility..
Where we are conversing with a passenger, we have the prior knowledge of who they are, and that can help with the scope of the conversation. We are aware of the same sensual queues of the motion of the vehicle, and external activities. It is a common characteristic of conversations in cars that the driver's responses are often interrupted by short pauses, whilst concentration is directed to a driving manoeuvre or situation. As a sensitive passenger, you will share the same motion and visual queues as the driver and accept, understand and compensate for the interruptions in the conversation.
For a telephone conversation, the situation is more difficult. If we receive a call, at first we are not sure of who we will be talking too, nor the likely reason for the call. So we have to apply more mental effort to prepare for a 'blind' conversation. The two parties do not share the same motion or visual clues, and thus breaks in conversation are not so easily assimilated. Consequently, both parties have to work harder to keep the conversation flowing and relevant. The extra concentration required has to be to the detriment of a driver's ability to concentrate on driving. This is even more so if the driver has to hold the handset to his ear, which becomes a physical impediment, and introduces a dilemma of what to do with the handset if an manoeuvre is required.
I think it is wholly relevant that the use of hand held mobile phones has been outlawed. In my opinion they represent a significantly greater distraction to drivers than most other distraction issues.