We spent an afternoon here and had a very enjoyable time, though neither of us know too much about art. Their collections are extensive and run the gamut from Renaissance masters to more contemporary abstract pieces.
The museum and gardens are open 12pm - 6pm daily except Tuesdays. We spent a Sunday afternoon there... not crowded at all, easy parking and admission price ($6) was reasonable. Seniors are half-price and students with a valid ID are free.
The grounds are well-manicured both front and back and were an idyllic place to just relax. In the far back portion of the garden, the 210 Fwy and Colorado Blvd intruded somewhat on the mood; but if you come in just a few yards closer to the main house, the noise level drops off considerably. There is a cafe outside in the back garden - basic fare but all the essentials are there.
When we went, there was an exhibition of Indian art, which was very interesting and eye-opening. In addition, the home collection was very extensive. We found that we enjoyed the classical paintings more than the more contemporary art. They have available an audio commentary for rent when you get there. Maybe next time, we'll try it.
Overall, an excellent time. If you're visiting, and only get to go there once, spend more than an afternoon - you're really selling yourself and the museum short if you spend less than a day. Evening dining is close by at Old Town Pasadena, so make a day-trip out of it.