Tom's isn't the cleanest or fanciest place by any stretch of the imagination. It's more like an old rundown diner ... but it's a cheap quick place to get a delicious breakfast ($5 including tax!).
It's across the street from CAMH so some of the people hanging around are kind of sketchy ... but just politely decline whatever they want (asking money, selling you a radio, bumming a cigarette, etc.) and you'll be fine.