I love Toronto Island. The best thing to do here is rent a bike, even one of those silly 4 seaters with a group, and just go exploring. I like packing a lunch, finding a quiet spot to eat it, and just enjoying the nice weather.
This place is really family friendly - you may want to avoid the nude beach, but you won't see it if you don't want to.
This is a great way to spend the day - the only thing is I wish there were more choices as far as restaurants go. I'm not a big fan of the ones there.