We were in the Ile d'Oleron last June, the earlier part and it was far from over crowded then thought undoubtedly it bulks up on week ends. In June I feel its not going too be bad.
The cycle routes are a great plus for us, as well as it being flat but flat, so effortless cycling on a calm day.
Time you access to either totally miss or be counter to the day trippers; like don't go on on Friday night Saturday morning.
We stayed on an ACSI fixed price site, Le Suroit, not far from good shopping and a nice fishing port. Would use the site again but its not exciting; tend to think most sites on what is only sand tend to trier easily but I would not like to leave the impression this is a bad site, its not. Free WiFi around reception.
What I would do a bit of homework on though is, how did these two Islands fare in the recent storms? I think very badly from UK TV coverage, so would want to check if its been well cleaned up.