A castle in Toronto? I'm there!
I think you have to take one thing into consideration when you go here... Castles (the pure definition of one) originated in the 9th and 10th Century and were fortifications for lords and nobles. To call the Casa Loma a castle is too grandiose a title. Its more, like Elvis A. stated, this [is] one man's ostentatious monument to himself! So, this is more of a castle-like palace.
If you keep the fact that Casa Loma is an elaborate home built in the early 1900s that sent a man into financial ruin it will help you to enjoy your visit here. I went there with that in mind (and a discounted ticket from City Pass) and had a nice time! I also had an encounter with a ghost, but that was just me.
Not-so-secret passageways lead you around in circles here. An unfinished basement with a spooky underground swimming pool is like a scene out of the Titanic, made me feel like everyone on board is going down... I imagined that Sir Henry Pellatt himself looked out at his unfinished pool and often felt like he was drowning, in debt. It was an eerie feeling. Walking back towards the main house from the pool, we felt a weird creepy feeling. It got stronger and stronger until, eventually we found ourselves running back to the main house!
I can't guarantee you will get the same spooky feelings we did- but Casa Loma was one of the more interesting places I visited in Toronto, and I would recommend it!