I used to purchase all my seafood from the St. Lawrence Market, but then I discovered Diana's.
Not the most convenient spot for downtowners but their prices and selection (especially for oysters) can't be beat. It's definitely my "go-to" for oysters of all different varieties; they seem to carry a lot of varieties that can't be found elsewhere. Also the place to go if you're looking for unusual/hard-to-find seafood items like geoduck, live sea urchin, live pink sea scallops, rock shrimp, etc.
For those of you looking for whole fish, you can pick one out and have it scaled and cleaned on site (free of charge I believe).
Everything seems very fresh and I've had no fear eating raw the sushi-grade fish I've purchased from Diana's (wish I could say the same of some of the sushi I've had in the city!).
Definitely worth the trip to Scarborough!