I lived in Kibuye for two years, so I'm a bit biased, but this is an amazing place. Kibuye sits straight west of Kigali (about 2 hours and 15 minutes in a direct minibus) on the edge of Lake Kivu. On the road, you cross the continental divide - the rivers in Kigali flow toward the Nile, and the rivers in Kibuye flow toward the Congo River. There are no taxis or minibuses in Kibuye, but you can hire a motorbike or a bicycle to take you around - tell them when to meet you if you want them to come back later. You should pay about 200-300 Rwandese francs per ride on a motorbike, 50-100 on a bicycle. The town is also very walkable, although you should go with a friend at night, as always.
There are several places to stay in Kibuye, which is built on a loop of paved road. To the right off the only roundabout is the Home St. Jean, behind the church at the top of the hill. This is about $5 per night but a bit damp and close. To the left off the roundabout are several small places - ask around. One is the Auberge La Nature, up the hill behind the petrol station. This is about $5 per night and you can eat there - but I can't vouch for the perfect cleanliness of the rooms. The two more expensive places are on the other side of the loop. For the Centre Bethanie (run by the Presbyterian Church), go straight at the roadabout and stay on the paved road where it branches to the right at the hospital. At the top of the hill, stay straight on the dirt road next to the big electricity tower. Bethanie is at the end of the dirt road. For the Hotel Golf Eden Rock, go right at the roadabout, past the Catholic Church and a ways down the hill. There is a sign and parking lot on your left. The Kibuye Guesthouse of guidebook fame is closed pending remodeling and reopening as something horrible and large.
Kibuye is a great place to sit and relax. The beach was at the guesthouse and so currently unavailable, but you can hire a boat (about $10 from Bethanie or in front of the Eden Rock) and go out to the Island called Amahoro (Peace). It has volleyball, little huts, and some sandy areas. Swimming is possible here or near Bethanie - guidebooks and doctors in the US may tell you not to swim, but a good friend of mine there was a doctor and told me that there is no bilharzia in the Kibuye area. Swim freely. There are also nice trails heading off everywhere - take one that looks like it heads away from town and walk for a while. The area is very safe. If you are with a group, find a small restaurant (like Chez Mama Manzi, near Auberge La Nature behind the petrol station) and order gisafuriya, a goat and banana dish with peanut stew. Classic Rwandan fare. Otherwise, goat brochettes are the cheap, filling way to go.