I've come to Toby's twice. Loved it the first time, unimpressed the second. Both times were very crowded but on our first visit we arrived at about 6 and had probably an hour wait, the second time we arrived around 7:30 and had to wait about 2 hours for our food. The first time the fish (we shared a few different kinds) was fantastic (like best in town) the second time it was no better than what you'd get anywhere else. The first time the servers/bartenders were friendly, if busy, the second time they really seemed like they wished we (or they) weren't there.
I felt like the strange model of hailing down a server and ordering when you put your name on the seating list was kind of quirky and fun the first time we went. On the second trip it just seemed like a big pain in the ---.
Definitely won't go back later than 6 o'clock on a Friday. May not go back at all.