Since they have built the tunnel connecting Toronto to the island airport, travel has become even more convenient! I used to build in an extra 15 minutes for the ferry but now I can easily go city to island (or vice versa) in less than 5 minutes if I walk at a normal pace on the movators. If you plan to stand and take your time instead of walking, I'd recommend still planning a 10 minute buffer.
Note on the city side, there are only elevators and stairs (no escalators). On the airport side, there are escalators, elevators, and stairs. I'd recommend the elevators if you are pressed for time as there are 2 sets of escalators, which can take awhile.