This is a lovely place to visit anytime during the year. Like all good gardens, it has the hardscape and supporting structures to keep it interesting, even without tons of flowers, etc. However, it's even better starting in mid-spring when the perennials start coming through.
There is a really nice visitor center as a starting point, and then the grounds spread out from there. There are so many little garden 'rooms' with water features and other surprises- like a lovely large bird cage.
You can bring your own picnic inside as well.
One note: on Mother's Day, they offer free entry and a free plant to all moms. Therefore, it tends to get MOBBED.