Even though the location is a bit of a busy area, i've always managed to have a nice intimate dinner at Spring Rolls on Yonge Street.
The atmosphere is lively, the decor is charming, the restaurant itself is clean, and the food is quite good. The waitstaff could be a bit friendlier, but i've never had any bad experiences with them. It is a bit of a pain that the restroom is on the ground floor, but otherwise, Spring Rolls provides a quality experience, even for a New Yorker who only occasionally visits Toronto.
I HIGHLY recommend the seafood fried rice - i could eat that ALL DAY LONG. The drinks are also pretty tasty.