Such a lovely place, it makes sense that it was the setting for the movie 'The Scarlett Letter' with Demi Moore. Rent a copy just to see the fabulous scenery! Often the harbour has beautiful sail boats that have come up from the Caribbean for a nice, cool summer. Go out of your way to get to Shelbourne. It's well worth the effort.
One of the finest natural small boat harbours I've ever seen. A safe haven in rough weather. Of course in colonial days, the largest ships in the realm could come in, but they were tiny by modern standards.