Annapolis has a harbour full of sailing boats, infact it's a hugely popular thing in Annapolis. You can take several boat trips from the Quay or you can just walk around looking at the different types of boat. (This seems to be what the locals do).
Also down by the water are several crafty shops, bars and the local speciality of seafood restaurants. If you want to just wander and stop to browse or have a drink now and again, this is the place to go.