I had a pleasant experience renting bikes from this place. It was easy and affordable.
We rented two single bikes to cruise around the island. For $8 for the first hour and $4 per half hour increment thereafter, you get a simple beach cruiser type bike. It did not come with a lock or a basket. You deposit $10 per rental with an ID, and either pay additional charges or get money back when you return the bike. In addition to single bikes, they also rent tandem, and quad bikes.
It's located in the back of the island, so once you get off the ferry at Centre Island, you walk an additional 10 minutes or so to get to the rental location. Looks like the shack is run by an Asian family, and service they may be abrupt at times.
PS. It's cheaper to rent once of the city bikes, I think it was only $7 for the entire day, but then you have to transport it onto the ferry against heavy pedestrian traffic. So it may be best to rent here.