The Mattress Factory is the definition of trippy. When I went in the middle of a weekday, it was deserted and walking around the installations by myself only enhanced the experience. Many people have mentioned the Yayoi Kusama pieces, and "Repetitive Vision" is by far the most memorable permanent exhibit in the museum. I recommend starting off with the James Turrell's light installations - there's no better way to begin your Mattress Factory experience. Don't forget to check out the work at the Annex as well, which is just down the street on Monterey.
Bonus: parking is free and on Thursdays, admission is half-price. If you're a student, this means that you can get in for just $3. I pay more than that for a latte, people. One more reason why the MF can't be missed.