As other reviewers have stated, Penn Avenue Fish Company has the best fish in the city. Bearing that in mind, it is also some of the most expensive fish in the city. I go here for fish when I need something perfect or want something special or if I'm making sushi at home (never skimp for sushi.)
They also have a casual order-at-the-counter lunchtime business. Once again, the prices are higher than a normal "fast food" lunch option, but the quality is top-notch. I like the fish tacos - 2 tacos make a generous lunch for me.