There are a lot of unique spices here that are difficult to impossible to find anywhere else locally (save, I expect, for the WSM or the other Penzey's shop in Ohio City). If you're looking for celery seed or Sichuan peppercorns and are striking out at Whole Foods, you'll be able to find what you need at Penzey's.
The spices I've purchased at Penzey's are good quality but a bit expensive. I'm continually disappointed that they don't really have anything in the way of organic offerings: while I know it's not always possible to source everything as organic, there's really no excuse for such an upmarket store to not even try.
The store itself is Spartan, and not in a good way: there's little in terms of decoration, with some old burlap bags tacked to the wall and plain wooden shelving. It looks a lot like one of those temporary holiday mall storefronts, where everything has been deliberately kept dull to make it easier to take the whole thing apart again in a few weeks. Again, it's just not what you'd expect for an expensive store in one of the most high-end shopping malls in the city.
My final annoyance with Penzey's is the short hours: they close at 6pm! I often feel the urge to buy some exotic spice after work for that day's dinner, and the early closing time makes it next to impossible. Maybe this says more about my planning skills than anything, but still...