Just go. Really. It's a quintessential Madison experience. My world halts on Saturday morning for this market.
There's lots to see and explore, but I (highly) recommend stopping at one of the bakery stalls (Stella's is right near the intersection of State St. and the Capitol Square) and picking up a pastry or loaf of bread to nibble on while you walk :D