Show up hungry. Seriously.
Skip a meal or two if you have to - you won't regret it. The Ferry Building is a historic city landmark that survived every major earthquake it met (1906 and Loma Prieta), and ferries still operate from the building (albeit fewer than in the days when the only way to get into San Francisco was by water). The view from the water side of the building is fantastic, but it's what's inside that draws most visitors to the Ferry Building, and rightly so.
Inside the building are more than 40 shops, restaurants and cafes, mostly selling gourmet food. There are two stores focused on olive oil, three fishmongers, a couple of butchers, a patisserie, a confectioner, half a dozen fruit and vegetable markets, a store that sells nothing but exotic mushrooms and, conveniently located right next to each other, a specialty cheese shop and a bakery selling fresh breads, including the local sourdough - a must-try for visitors to San Francisco. And, in case all the gourmet ingredients are particularly inspiring, there's a Sur La Table for high-end kitchen equipment. Most people will pick up the makings of a picnic here, but you can also get takeaway prepared Vietnamese, Japanese, Mexican, French, and deli meals, or eat in one of three sit-down restaurants. If I had any nerve at all, I'd have put the $73 caviar plate at Tsar Nicoulai on my expense account, but I stuck with an under-$10 sandwich from Lulu Petite instead, and was well satisfied. I finished off my lunch at the Ferry Building with two tiny squares of Scharffen Berger chocolate. Out-of-towners think "Ghiradelli" when they think of San Francisco, but Scharffen Berger is also a local chocolate-maker, and their chocolate is the better of the two by a wide margin. If you can't make it out to Berkeley for a factory tour, the next best thing is a visit to the retail store in the Ferry Building.
Before you plunk down money for any food at the Ferry Building, pop into every store that strikes your fancy - most of them have samples out and you can practically make a meal of olive oil-drenched cubes of sourdough, smears of spreads and dips on crackers, and finely-portioned appetizers and slices of exquisite cheese. Finish off your cheapskate tour with a tiny spoonful of gelato, and don't you dare be ashamed!
There are free walking tours of the building (with even more samples!) at noon on Saturdays, Sundays and Tuesdays, and a farmer's market on Saturdays - that's when locals pack the place, so if crowds aren't your thing, save your visit for a weekday afternoon. (Fresh organic produce isn't so hard to find in SF as in other parts of the country, so don't be afraid of missing out.)