Out of all the things I did while visiting Toronto, this was my favorite! It is a bit pricey, but there is a LOT to see.
I went on a Thursday afternoon, but it was still very crowded. The entire house is well kept, interesting and simply amazing. The gardens are also very nice! Make sure you visit the tunnels that lead to the garage and stables.
All in all, a very cool piece of history worth the time and effort! :)