Between the Rivoli and the Queen Mother, Im more partial to the former although neither is really my bag.
I warn you now, it gets drafty in there in the winter, but the back patio is sweet in the summer. It's big brunch spot and it's been around for a long time so all the old Queen Street people talk about it like they still go every day.
Locals call it the Queen Mum and they have great booths for an intimate lunch but not a great bar scene if you're looking for something to do later on. Much more of a day spot than the Riv.