Dufflet is a Toronto institution--even if you've never gone into the store, you've probably had a Dufflet pastry at some point (a catered work lunch, a friend's baby shower, a pick-me-up as you leave Whole Foods).
The shop itself isn't huge, but there's charming art on the walls and it's easy to strike up a conversation with other people in the shop. I chose a day-old molten lava cake (day-old pastries are discounted, and I am totally a sucker for that sort of thing) that was absolutely delicious. It's a nice place to stop in while strolling that section of Queen, and if all you know of Dufflet are the pastries in the case at Whole Foods or that cake you had that one time, it's worth a visit.