More estate than castle, Casa Loma is located not too far from downtown Toronto, but just be sure you're inputting the right address on Uber/Lyft as the official location/address is literally someone's backyard.
The history here is pretty interesting and they've got the usual roped-off rooms you see at other historicaly sites. What I did think was uniquing/interesting were the bell towers and the underground passage. Don't forget to check those out.