St Sophia's Cathedral is an amazing place, ancient and beautiful.
The spooky underground Caves Monastery is fantastic.
Quick Tips:
This isn't a safe place. Of all the places I've ever been to, I felt least safe in this city. Be very, very careful here. We didn't go out after 7pm once, as we just didn't feel safe enough here. You also have to be careful in your hotel.
Best Way To Get Around:
I wouldn't hire a car here, there's no real point and it makes you a target for criminals and police alike. Public transport is cheap, and frequent. Taxis are reasonably cheap, but as in the rest of the ex-USSR, keep your wits about you.