The Intercontinental has a location that can't be beat. It's convenient to virtually everything downtown and incidentally has a very nice pool (with bonus salt water hot tub option!). Additionally it's connected to the Convention Center so if you're there for a convention then it's sort of a no-brainer. Otherwise it's very expensive and the amenities honestly aren't really what you'd expect for the price.
To describe the Intercontinental experience, picture a Holiday Inn with a great view of Toronto and multiply the price by two or three. That said, in the off-season it's pretty easy to grab a room on Priceline and then it's definitely worth the money! Otherwise the underwhelming amenities combined with the price, even considering the location, land three stars for the Intercontinental.