Pacific Avenue is what locals mean when they say "downtown". It is a strip of street, about 8 blocks long, lined with stores, resturants, and a few movie theaters. It is the city's center of activity. Start at either end, and just walk down Pacific' this can be an evening. There are lots of interesting stores to go into, and people to watch.
If you want to take the bus, must buses will take you right there, because the bus station is at one end of Pacific. You can also drive there fairly easily, because the city has several free parking lots directly on, and nearby.
Also there are some interesting things if you go a block or two off of Pacific, such as the Nicleodian theater, and a variety of cafes and resturants. All around this is a very good place to go, and probably the best area of Santa Cruz.