It's small, and I was in and out of there in less than an hour. I started on the second floor and immediately thought, "Wait, that's it?" The third floor was more of the same.... sparseness. The top floor is more expansive and houses its main gallery. I enjoyed the Matisse collection of some of his book illustrations.
You won't spend a whole day here. Nor will you spend a whole afternoon here. My suggestion is that you couple this visit with the Mint museum next door.
Take an obligatory picture of the whimsical firebird sculpture in front of the building.