Bareburger is very family friendly (if you go at the down times). Their collard green wrap is a great gluten-free alternative; My husband often orders it just because he likes the taste (although he says their buns are pretty perfect). Their pickle plate is a good starter, and the Be My Burger is an excellent option for those who have allergies. Hands down, they have the best sweet potato fries in Toronto.
The only thing that might deter folks is that the food takes time. It's not the service that's slow (they are all helpful and considerate, for the most part), it's just that the kitchen is kind of glacial. So, if you're eating in or doing take out, expect to wait in West Coast time.