Stone Town is a must! But don't forget to take a city map with you, well, even with a map you will get lost in those narrow streets!
I visited al the small remains of the cultural past of Zanzibar on my own (slave cave, some palaces and Persian baths). It's very cheap, and it only takes half a day!
Quick Tips:
Don't do one of those stupid spice tours! Stay clear of the all-inclusive hotels, they are packed with arrogant Italians!
Best Way To Get Around:
Arrange a taxi yourself (not by the hotel) and just go where you want to go. It's cheap, easy, fast and you can do you just what you want to do!!!