NYC to Montreal: wait time was an hour + at a 4pm on a Thursday. First line is so they check your car, second line (longer) is for passport check. Two lanes open. WUT. On top of that the Canadian agent gave us some mad 'tude.
Montreal to NYC: A cluster mess of lines. They diverged at so many different points, so it was hard to tell which were actually open at the time. Went much faster - only a 20-30 min wait (then again, there were at least 5 lanes open). There was no car check, which was a bit confusing.
It's a border. Be prepared for lines that don't move and people who hate their jobs.